Discrete Time Markov Chains Discrete-Time Markov Chains Books - - - PDF document

SLIDE 1

Markov Processes and Applications Markov Processes and Applications

D k h

• Discrete-Time Markov Chains
• Continuous Time Markov Chains
• Continuous-Time Markov Chains
• Applications

Applications

– Queuing theory – Performance analysis

ΠΜΣ524: Μοντελοποίηση και Ανάλυση Απόδοσης Δικτύων (Ι. Σταυρακάκης / Α. Παναγάκης - ΕΚΠΑ) 1

Discrete Time Markov Chains Discrete-Time Markov Chains

Books

• Introduction to Stochastic Processes (Erhan Cinlar), Chap. 5, 6

Introduction to Probability Models (Sheldon Ross) Chap 4

• Introduction to Probability Models (Sheldon Ross), Chap. 4
• Performance Analysis of Communications Networks and Systems

(Piet Van Mieghem), Chap. 9, 11

• Markov Chains (J.R. Norris), Chap. 1
• Discrete Stochastic Processes (R. Gallager), Chap. 4

Elementary Probability for Applications (Rick Durrett) Chap 5

• Elementary Probability for Applications (Rick Durrett), Chap. 5
• Introduction to Probability, D. Bertsekas & J. Tsitsiklis, Chap. 6

ΠΜΣ524: Μοντελοποίηση και Ανάλυση Απόδοσης Δικτύων (Ι. Σταυρακάκης / Α. Παναγάκης - ΕΚΠΑ) 2

SLIDE 2

th order pdf of some stoc. proc. { } is given by

t

n X INTRODUCTION :

1 2 1 1 1 2 3 1 2 1 1

p p { } g y ( , ,..., ) ( | , ,..., ) ( | , ,..., ) ... ( | ) ( )

n n n n n n n

t t t t t t t t t t t t t t t

f x x x f x x x x f x x x x f x x f x

− − − −

=

2 1 1

very difficult to have it in general If { } is

t t t t

X

• an indep. process:

1 2 1 1

( , ,..., ) ( ) ( )... ( ) If { } is a process with indep. increments:

n n n

t t t t t t t

f x x x f x f x f x X

−

=

• 1

2 1 2 1 1

( , ,..., ) ( ) ( )... ( ) : First order pdf's are sufficient for a

n n n

t t t t t t t t

f x x x f x f x x f x x

−

= − − Note bove special cases If { } is a process whose evolution beyond is (probabilistically)

t

X t

• 1

2 1 2

completely determined by and is indep. of , , given , then: ( , ,..., ) ( | )... (

n n n

t t t t t t t t t

x x t t x f x x x f x x f x

−

< =

1 1

| ) ( )

t t

x f x

3

This is a Markov process ( th order pdf simplified) n

ΠΜΣ524: Μοντελοποίηση και Ανάλυση Απόδοσης Δικτύων (Ι. Σταυρακάκης / Α. Παναγάκης - ΕΚΠΑ)

Definition of a Markov Process (MP) Definition of a Markov Process (MP)

called is set a from values s that take } ; { proc stoch A ∈ E I t X countable) ( ) | ( ) | ( : iff (MP) Process Markov a called is set a from values s that take } ; { proc. stoch. A = ∈

t

E x x P x x x f E I t X e) uncountabl ( ) | ( ) | (

• r

countable) ( ) | ( ) ,..., | (

1 1 1 − −

=

n n n n

t t t t t

E x x f x x x f E x x P x x x f . all and ... all and all for e) uncountabl ( ) | ( ) ,..., | (

1 1 1

2 1

− −

> < < < =

n n n n n

n t t t t t t

n t t t x E x x f x x x f } ,..., { past" " the

• f

indep. is state next" " The :

2 1 − n n

t t t

x x x Notice known. is present" " that the provided

ΠΜΣ524: Μοντελοποίηση και Ανάλυση Απόδοσης Δικτύων (Ι. Σταυρακάκης / Α. Παναγάκης - ΕΚΠΑ) 4

SLIDE 3

Definition of a Markov Chain (MC)

i b bilit t iti b th d ib d i d MC a called is MP a then countable is and countable is If MP) value

• discrete

& time

• (Discrete

E I MC) the

• f

space

• (state

,...} 2 , 1 , { Assume MC). s homogeneou

• time

a for

• f

(indep. , } | { ) , ( : ies probabilit n transitio by the described is and

1

E n E j i i X j X P j i p

n n

= ∈ = = =

+

) p ( , } , , { ) g ( p ⎤ ⎡ : matrix Transition ... ) , 1 ( ... ) 1 , 1 ( ) , 1 ( ... ) , ( ... ) 1 , ( ) , ( n P P P n P P P P ⎥ ⎥ ⎥ ⎤ ⎢ ⎢ ⎢ ⎡ = M M M ... ) , ( ... ) 1 , ( ) , ( n n P n P n P P ⎥ ⎥ ⎥ ⎥ ⎦ ⎢ ⎢ ⎢ ⎢ ⎣ = M M M M M M d constructe be may MC a matrix) (stoch. given a For matrix) c (stochasti i , 1 ) , ( negative,

• non

is P j i P P

j

∑

∀ =

ΠΜΣ524: Μοντελοποίηση και Ανάλυση Απόδοσης Δικτύων (Ι. Σταυρακάκης / Α. Παναγάκης - ΕΚΠΑ) 5

d co st ucte be ay C a at ) (stoc . g ve a

• then

} { ) ( s t

• n

PMF a is If E i i X P i E ∈ = = : rule Chain π π

1 1 2 2 1 1

) , ( )... , ( ) ( } ,..., , , { then , , } { ) ( s.t.

• n

PMF a is If

n n n n

E i i i N n i i P i i P i i X i X i X i X P E i i X P i E

−

∈ ∈ ∀ = = = = = ∈ π π π

1

,..., , ,

n

k E i i i N n ∀ ∈ ∈ ∀ N : s transition step

• k

) , ( } | { ,

k n k n

j i P i X j X P k

+

= = = ∈ ∀ N . matrix n transitio the

• f

power th the

• f

entry ) , ( the is ) , ( ; , ,

k

P k j i j i P k E j i ∈ ∀ ∈ ∀ N

2 2 1 1 3

) , ( ) , ( ) , ( } | { ) iterations through (general 3 For : Proof

n n

j l P l l P l i P i X j X P n k

∑ ∑

+

= = = = 4 4 4 4 4 3 4 4 4 4 4 2 1 4 4 4 3 4 4 4 2 1

) , (

3 1 1 2 2

E l j l P E l

∑ ∑

∈ ∈ 6 ) , (

3

j i P ΠΜΣ524: Μοντελοποίηση και Ανάλυση Απόδοσης Δικτύων (Ι. Σταυρακάκης / Α. Παναγάκης - ΕΚΠΑ)

SLIDE 4

Chapman Kolmogorov Equations : From previous, ( , ) ( , ) ( , ) ,

+ ∈

= ∈

∑

m n m n k E

P i j P i k P k j i j E In order for { } to be in after steps and starting from , +

n

X j m n i it will have to be in some after steps and move then t k m

• in

the remaining steps. j n

7 ΠΜΣ524: Μοντελοποίηση και Ανάλυση Απόδοσης Δικτύων (Ι. Σταυρακάκης / Α. Παναγάκης - ΕΚΠΑ) siklis ekas & Tsits Bertse ΠΜΣ524: Μοντελοποίηση και Ανάλυση Απόδοσης Δικτύων (Ι. Σταυρακάκης / Α. Παναγάκης - ΕΚΠΑ) 8

SLIDE 5

# of successes in Bernoulli process { } f i i l Example : { ; 0} , # of successes in trials , ,

• indep. Bernoulli, {

1} ≥ = = ≥ = =

∑

n n n n i i i

N n N n N Y n Y P Y p

1 1

Notice: evolution of { } beyond does not depe

= + +

= + ⇒

∑

i n n n n

N N Y N n

1

nd on { } (given ) and thus { } is a M C

− n

N N N does not depe nd on { } (given ) and thus { } is a M.C. { | } { | }

=

= = = −

i i n n

N N N P N j N N N P Y j N N N N

1 1 1 1

{ | , ,..., } { | , ,..., } ... if 1

+ +

= = = − ⎡ ⎤ = + ⎧ ⎢ ⎥ ⎪ ⎢ ⎥

n n n n n n

P N j N N N P Y j N N N N q p p j N q p ... 1 if and ...

• therwise

⎪ ⎢ ⎥ = = − = = ⎨ ⎢ ⎥ ⎪ ⎢ ⎥ ⎩ ⎣ ⎦ M

n

q p q p j N P q p Notice: { } is ⎣ ⎦ M

n

N a special M.C. whose increment is indep. both from present and past (process with indep. increments)

9

p p (p p )

ΠΜΣ524: Μοντελοποίηση και Ανάλυση Απόδοσης Δικτύων (Ι. Σταυρακάκης / Α. Παναγάκης - ΕΚΠΑ)

: Sum of i.i.d. RV's with PMF { ; 0,1,2,...}

k

p k = Example

1 2

... 1

n n

n X Y Y Y n = ⎧ = ⎨ + + + ≥ ⎩ 1

2 n

X X Y ⎩ = +

1 1 1 1

{ | ,..., } { | ,..., } Th { } i M C i h ( ) { | }

n

n n n n n n n n j X

X X Y P X j X X P Y j X X X p X P P X X

+ + + + −

= + = = = − =

1 1 2 3

Thus { } is a M.C. with ( , ) { | }

n n n j i

X P i j P X j X i p p p p p

+ −

= = = = ... ⎡ ⎤ ⎢ ⎥ P =

1 2 1

... ... p p p p p ⎢ ⎥ ⎢ ⎥ ⎢ ⎥ ⎢ ⎥

1

... p p p ⎢ ⎥ ⎢ ⎥ ⎢ ⎥ ⎣ ⎦ M M M M O

10

⎣ ⎦ M M M M O

ΠΜΣ524: Μοντελοποίηση και Ανάλυση Απόδοσης Δικτύων (Ι. Σταυρακάκης / Α. Παναγάκης - ΕΚΠΑ)

SLIDE 6

2 1 ) ( ith i i d t trials Independen : k k X X Example ) ( } { } ,..., | { ,... 2 , 1 , , ) ( with i.i.d. ,... ,

1 1 1

j j X P X X j X P k k X X

n n n

= = = = =

+ +

π π ) 1 ( ) ( M.C. a is } {X n ⎥ ⎤ ⎢ ⎡ L π π ) 1 ( ) ( P ⎥ ⎥ ⎥ ⎥ ⎢ ⎢ ⎢ ⎢ = M M L π π ) 1 ( ) ( ⎥ ⎥ ⎥ ⎦ ⎢ ⎢ ⎢ ⎣ O M M L π π ) i i d th id ti l ll h If ( 1 and identical are rows that Notice X X P m P Pm ≥ ∀ = ⎥ ⎦ ⎢ ⎣ O M M ) i.i.d. are ,... , then identical rows all has If (

1 0 X

X P

11 ΠΜΣ524: Μοντελοποίηση και Ανάλυση Απόδοσης Δικτύων (Ι. Σταυρακάκης / Α. Παναγάκης - ΕΚΠΑ)

} { ith } 4 3 2 1 { i i d } { Y Y E l M.C. a is } { , 5) (modulo } , , , , { with } 4 , 3 , 2 , 1 , { i.i.d. are } { :

1 1 4 3 2 1

+ = ∈

+ +

X Y X X p p p p p Y Y

n n n n n n

Example matrix) (stoch 1 rows

4 3 2 1

∑

⎥ ⎥ ⎤ ⎢ ⎢ ⎡ p p p p p p p p p p (here) 1 columns matrix) (stoch. 1 rows

2 1 4 3 3 2 1 4

∑ ∑

= = ⎥ ⎥ ⎥ ⎥ ⎢ ⎢ ⎢ ⎢ = p p p p p p p p p p P matrix) stochastic

• (double

4 3 2 1 1 4 3 2

⎥ ⎥ ⎥ ⎦ ⎢ ⎢ ⎢ ⎣ p p p p p p p p p p

4 3 2 1

⎦ ⎣ p p p p p

12 ΠΜΣ524: Μοντελοποίηση και Ανάλυση Απόδοσης Δικτύων (Ι. Σταυρακάκης / Α. Παναγάκης - ΕΚΠΑ)

SLIDE 7

: Remaining lifetime Example : Remaining lifetime An equipment is replaced by an identical as soon as it fails Example Pr{a new equip. lasts for time units} 1,2,...

k

p k k = = remaining lifetime of equip. at time ( ) 1 if (

n

X n X X ω = ) 1 ω ≥ ⎧

1

( ) 1 if ( ( )

n n n

X X X ω ω

+

− =

1

) 1 ( ) 1 if ( )

n n

Z X ω ω ω

+

≥ ⎧ ⎨ − = ⎩

1 1

( ) is the lifetime of equip. installed at time It is independent of , ,...,

n n

Z n X X X ω

+ 1

p , , , is a M.C.

n n

X

13 ΠΜΣ524: Μοντελοποίηση και Ανάλυση Απόδοσης Δικτύων (Ι. Σταυρακάκης / Α. Παναγάκης - ΕΚΠΑ)

1: ( ) { | } { 1 | }

• ≥

= = = = − = = i P i j P X j X i P X j X i

1

( , ) { | } { 1 | } 1 if 1 { 1| } if 1

+

= = = = = = = − ⎧ = = + = = ⎨ ≠ ⎩

n n n n n n

P i j P X j X i P X j X i j i P X j X i j i if 1 0 : ≠ − ⎩

• =

n n

j i i

1 1 1 1

(0, ) { | 0} { 1 | 0} { 1}

+ + + +

= = = = − = = = = + =

n n n n n j

P j P X j X P Z j X P Z j p

1 2 3 4

1 ⎡ ⎤ ⎢ ⎥ L p p p p 1 1 1 ⎢ ⎥ ⎢ ⎥ ⎢ ⎥ = ⎢ ⎥ L L P 1 ⎢ ⎥ ⎢ ⎥ ⎢ ⎥ ⎣ ⎦ L M M M M O

14 ΠΜΣ524: Μοντελοποίηση και Ανάλυση Απόδοσης Δικτύων (Ι. Σταυρακάκης / Α. Παναγάκης - ΕΚΠΑ)

SLIDE 8

1

: (conditional indep. of future from past given present) Let be a bounded function of , ,... . Then

n n

Y X X + Theorem

1

E{Y| , ,..., } E{Y| }

n n

X X X X =

1 1

E{ ,... | } E{ ,... | }

n n n

f(X ,X )X i f(X ,X )X i

+

= = = Proposition : Corol : a bounded function on ... f E E × × lary

1 1 1

Let E{ ,... | }. Then E{ ,... | , ,..., }

n n n n

g(i) f(X ,X )X i n N f(X ,X )X X X g(X )

+

= = ∀ ∈ =

1 1

{ | }

n n n n

f( ) g( )

+ 15 ΠΜΣ524: Μοντελοποίηση και Ανάλυση Απόδοσης Δικτύων (Ι. Σταυρακάκης / Α. Παναγάκης - ΕΚΠΑ)

: (conditional indep. of future from past given present) Theorem

1 1

( p p g p ) Let be a bounded function of , ,... . Then E{Y| , ,..., } E{Y| }

n n n n

Y X X X X X X

+

=

1

{ | , , , } { | }

n n ΠΜΣ524: Μοντελοποίηση και Ανάλυση Απόδοσης Δικτύων (Ι. Σταυρακάκης / Α. Παναγάκης - ΕΚΠΑ) 16

SLIDE 9

Proposition :

1 1

E{ ,... | } E{ ,... | }

n n n

f(X ,X )X i f(X ,X )X i

+

= = =

ΠΜΣ524: Μοντελοποίηση και Ανάλυση Απόδοσης Δικτύων (Ι. Σταυρακάκης / Α. Παναγάκης - ΕΚΠΑ) 17

: a bounded function on ... f E E × × Corollary

1 1 1

Let E{ ,... | }. Then E{ ,... | , ,..., }

n n n n

g(i) f(X ,X )X i n N f(X ,X )X X X g(X )

+

= = ∀ ∈ =

ΠΜΣ524: Μοντελοποίηση και Ανάλυση Απόδοσης Δικτύων (Ι. Σταυρακάκης / Α. Παναγάκης - ΕΚΠΑ) 18

SLIDE 10

time fixed for derived results Previous n∈N : Times Stopping instead? RV an is time if What time fixed for derived results Previous n∈N } ; { future the and } ; { past the , RV a for If

m m

T m X T m X T ≥ ≤

• .

at hold to said is property Markov strong then the , present given indep. lly conditiona are

T

T X ) ,..., , at looking by determined be can } { event the if time stopping a is ( true hold above then time, stopping a is If

1 n

X X X n T T T ≤

• )

g y } {

1 n 19 ΠΜΣ524: Μοντελοποίηση και Ανάλυση Απόδοσης Δικτύων (Ι. Σταυρακάκης / Α. Παναγάκης - ΕΚΠΑ)

T For any stopping time :

1 1

{ ( , ,...) | , } { ( , ,...) | } For ( ) { ( ) | }

T T n T T T

T E f X X X n T E f X X X g i E f X X X i

+ +

• ≤

=

• =

= For any stopping time :

1 1

For ( ) { ( , ,...) | } { ( , ,...) | ; } ( )

T T n T

g i E f X X X i E f X X X n T g X

+

• =

= ≤ = 1 if e g if ( )

m

a j f a a j E m N = ⎧ = ∈ ∈ ⎨

1

e.g., if ( , ,...) , 0 if { ( ,

m

f a a j E m N a j E f X X ∈ ∈ ⎨ ≠ ⎩

1

,...) | } { | } ( , )

m m

X i P X j X i P i j = = = = = { ( , f

1 1

, ) | } { | } ( , ) { ( , ,...) | , } { | ; }

m T T n T m n

j j E f X X X n T P X j X n T

+ +

≤ = = ≤ Strong Markov property at T: { | } ( )

m

P X j X T P X j

• ≤

20

{ | ; } ( , )

m T m n T

P X j X n T P X j

+

= ≤ =

ΠΜΣ524: Μοντελοποίηση και Ανάλυση Απόδοσης Δικτύων (Ι. Σταυρακάκης / Α. Παναγάκης - ΕΚΠΑ)

SLIDE 11

Visits to a state

{ } = ; ∈

n

X X n N MC, State space E , Transition matrix P . Notation: { } { } = | =

i

P A P A X i and [ ] [ ] = | =

i

E Y E Y X i Let ∈ j E , ω ∈Ω and Define: ( ) ω

j

N = total number of times state j appears in

1

( ) ( ) ω ω , , X X …. ( )

j

j pp

1

( ) ( ) , , ♣ ( ) ω < ∞

j

N , X eventually leaves state j never to return. ♣ ( ) ω = ∞

j

N , X visits j again and again. Let

1 2

( ) ( ) ω ω , , T T … the successive indices 1 ≥ n for which ( ) ω =

n

X j . ♣ If ∃ / n then

1 2 1

( ) ( ) ( ) ω ω ω = − = = ∞ L T T T

1 2 1

♣ If j appears a finite number of times m , then

1 2

( ) ( ) ( ) ( ) ω ω ω ω

+ +

− = − = = ∞ L

m m m m

T T T T ∀ ∈ n N { ( ) } ω ≤ T n is equivalent to j appears in { ( ) ( )} ω ω L X X at least m times ∀ ∈ n N , { ( ) } ω ≤

m

T n is equivalent to j appears in

1

{ ( ) ( )} ω ω , ,

n

X X at least m times.

m

T is a stopping time.

ΠΜΣ524: Μοντελοποίηση και Ανάλυση Απόδοσης Δικτύων (Ι. Σταυρακάκης / Α. Παναγάκης - ΕΚΠΑ) 21

Example

( ) 4 ( ) 6 ( ) 7 ( ) 9 T T T T

1 2 3 4

( ) 4 ( ) 6 ( ) 7 ( ) 9 ω ω ω ω = , = , = , = , T T T T … E j

ΠΜΣ524: Μοντελοποίηση και Ανάλυση Απόδοσης Δικτύων (Ι. Σταυρακάκης / Α. Παναγάκης - ΕΚΠΑ) 22

SLIDE 12

Proposition: 1 ∀ ∈ ≥ i E k m Proposition: 1 ∀ ∈ , , ≥ i E k m

{ }

1 1 1

{ } { } { }

+

= ∞ ⎧ − = , , = ⎨ = < ∞ ⎩

m i m m m j m

T P T T k T … T P T k T Computation of

1

{ } =

j

P T k . Let

1

( ) { } , = =

k i

F i j P T k 1 ( ) = ⇒ , =

k

k F i j

1 1

{ 1} { } ( ) = = = = ,

i i

P T P X j P i j 2 ( ) ≥ ⇒ , =

k

k F i j

1 1

{ }

−

≠ , , ≠ , = L

i k k

P X j X j X j { } { } ≠ ≠ |

∑

P X b P X j X j X j X b

=

1 2 1 1 { }

{ } { }

− ∈ −

= ≠ , , ≠ , = | =

∑

L

i i k k b E j P X

b P X j X j X j X b

=

1 1 2 1 { }

{ } { }

− − ∈ −

= ≠ , , ≠ , =

∑

L

i b k k b E j P X

b P X j X j X j Thus, ( ) 1 , = ⎧ ⎪ P i j k

1 { }

( ) ( ) ( ) 2

− ∈ −

⎪ , = ⎨ , , ≥ ⎪ ⎩ ∑

k k b E j

F i j P i b F b j k

ΠΜΣ524: Μοντελοποίηση και Ανάλυση Απόδοσης Δικτύων (Ι. Σταυρακάκης / Α. Παναγάκης - ΕΚΠΑ) 23

Example: Let 3 = j an the transition matrix

1 1/2 1/6 1/3 1/3 3/5 1/15

=

⎛ ⎞ ⎜ ⎟ ⎜ ⎟ ⎜ ⎟ ⎝ ⎠

P

1/3 3/5 1/15

⎜ ⎟ ⎝ ⎠ Find ( ) ( ) 1 2 3 = , , = , ,

k k

f i F i j i

• 1

= k . In this case

1

f is the 3rd column of matrix P.

1

Hence,

1 1

(1) (1 ) = , = f F j ,

1 1

(2) (2 ) 1 3 = , = / f F j ,

1 1

(3) (3 ) 1 15 = , = / f F j

1

(1 ) ( ) (1, )

−

, ,

⎛ ⎞ ⎜ ⎟ ⎛ ⎞ ⎜ ⎟

∑

k

P b F b j F j

1 ⎛ ⎞

• 2

≥ k . In this case

{ } 1 { } 1

1

(1, ) (2, ) (2 ) ( ) (3, ) (3 ) ( )

∈ − − ∈ − −

−

= , , , ,

⎛ ⎞ ⎜ ⎟ ⎜ ⎟ = = ⋅ ⎜ ⎟ ⎜ ⎟ ⎜ ⎟ ⎜ ⎟ ⎝ ⎠ ⎜ ⎟ ⎝ ⎠

∑ ∑

b E j k k k b E j k k

k k

F j F j P b F b j F j P b F b j

f Q f where 1 1/ 2 1/6 1/3 3/5 ⎛ ⎞ ⎜ ⎟ = ⎜ ⎟ ⎜ ⎟ ⎝ ⎠ Q

1 { } ∈ −

⎝ ⎠

∑

k b E j

After some algebra

1 2 3 4

1/3 1/18 1/108 1/ 648 ⎛ ⎞ ⎛ ⎞ ⎛ ⎞ ⎛ ⎞ ⎜ ⎟ ⎜ ⎟ ⎜ ⎟ ⎜ ⎟ = = = = ⎜ ⎟ ⎜ ⎟ ⎜ ⎟ ⎜ ⎟ ⎜ ⎟ ⎜ ⎟ ⎜ ⎟ ⎜ ⎟ L f f f f 1/15 1/5 1/30 1/180 ⎜ ⎟ ⎜ ⎟ ⎜ ⎟ ⎜ ⎟ ⎜ ⎟ ⎜ ⎟ ⎜ ⎟ ⎜ ⎟ ⎝ ⎠ ⎝ ⎠ ⎝ ⎠ ⎝ ⎠ and in general 1 1 ⎧ k

1 2

1 15 1 1 (1 3) (2 3) (3 3) 3 6 3 1 1 2 5 6 3

− −

⎧ = ⎪ ⎪ ⎛ ⎞ , = , , = , , = ⎨ ⎜ ⎟ ⎛ ⎞ ⎝ ⎠ ⎪ ≥ ⎜ ⎟ ⎪ ⎝ ⎠ ⎩

k k k k k

k F F F k

24

5 6 3 ⎪ ⎝ ⎠ ⎩

ΠΜΣ524: Μοντελοποίηση και Ανάλυση Απόδοσης Δικτύων (Ι. Σταυρακάκης / Α. Παναγάκης - ΕΚΠΑ)

SLIDE 13

1 2

1 1 15 1 1 (1 3) (2 3) (3 3) 3 6 3 1 1

− −

⎧ = ⎪ ⎪ ⎛ ⎞ , = , , = , , = ⎨ ⎜ ⎟ ⎛ ⎞ ⎝ ⎠ ⎪

k k k k k

k F F F 3 6 3 1 1 2 5 6 3 ⎛ ⎞ ⎝ ⎠ ⎪ ≥ ⎜ ⎟ ⎪ ⎝ ⎠ ⎩ k Now we can state: Now we can state:

• Starting at state 1, X never visits 3 with probability:

1 1

{ } 1 = +∞ = P T

• Starting at state 2 , X first visits 3 at k with probability:

1 1 1 3 6

( ) −

k

• Starting

at state 2 , X never visits 3 with probability:

1 3 1 1 2 1 2 1 3 6 5 1

{ } 1 { } 1 ( )

∞ − =

= +∞ = − < +∞ = − =

∑

k k

P T P T

• Starting

at state 3 X never visits 3 again with probability:

• Starting

at state 3, X never visits 3 again with probability:

52 3 1 3 1 75

{ } 1 { } = +∞ = − < +∞ = P T P T

ΠΜΣ524: Μοντελοποίηση και Ανάλυση Απόδοσης Δικτύων (Ι. Σταυρακάκης / Α. Παναγάκης - ΕΚΠΑ) 25

Now, for every , i j we define

1 1

( ) { } ( )

∞ =

, = < +∞ = ,

∑

i k k

F i j P T F i j ♣ ( ) , F i j expresses the probability: starting at i the MC will ever visit state j . ( ) ( ) ( ) ( ) , = , + , , , ∈

∑

F i j P i j P i b F b j i E

{ } ∈ −

∑

b E j

If by

j

N we denote the total number of visits to state j , then { } = =

j j

P N m

( )

1

( ) 1 ( )

−

, − ,

m

F j j F j j { } 1 ( ) − , = ⎧ ⎨ F i j m

and for ≠

i j , { } = =

i j

P N m

( )

1

( ) ( ) ( ) 1 ( ) 1 2

−

, ⎧ ⎨ , , − , = , , ⎩

m

j F i j F j j F j j m … >From the previous we obtain the Corollary: 1 ( ) 1 { } ( ) 1 , < ⎧ < +∞ = ⎨ = ⎩

j j

F j j P N F j j

ΠΜΣ524: Μοντελοποίηση και Ανάλυση Απόδοσης Δικτύων (Ι. Σταυρακάκης / Α. Παναγάκης - ΕΚΠΑ) 26

( ) 1 , = ⎩ F j j

SLIDE 14

Now, for every , i j we define

1

{ }

j j

P N m

∞

=

∑

1 1

( ) { } ( )

∞ =

, = < +∞ = ,

∑

i k k

F i j P T F i j

( )

1 1 1

( ) 1 ( )

m m m

F j j F j j

= ∞ − =

= , − ,

∑

1 1

m ∞

∑

♣ ( ) , F i j expresses the probability: starting at i the MC will ever visit state j . ( ) ( ) ( ) ( ) , = , + , , , ∈

∑

F i j P i j P i b F b j i E

( )

1 1 ( ) 1 1 ( ) F j j F j j = − , = − , , 1 1

m m

x x x

=

= < −

∑

{ } ∈ −

∑

b E j

If by

j

N we denote the total number of visits to state j , then { } = =

j j

P N m

( )

1

( ) 1 ( )

−

, − ,

m

F j j F j j { } P N 1 ( ) − , = ⎧ ⎨ F i j m

and for ≠

i j , { } = =

i j

P N m

( )

1

( ) ( ) ( ) 1 ( ) 1 2

−

, ⎧ ⎨ , , − , = , , ⎩

m

j F i j F j j F j j m … >From the previous we obtain the Corollary: 1 ( ) 1 { } ( ) 1 , < ⎧ < +∞ = ⎨ = ⎩

j j

F j j P N F j j

27

( ) 1 , = ⎩ F j j

ΠΜΣ524: Μοντελοποίηση και Ανάλυση Απόδοσης Δικτύων (Ι. Σταυρακάκης / Α. Παναγάκης - ΕΚΠΑ)

Now, for every , i j we define

{ }

j j

m P N m

∞

=

∑

1 1

( ) { } ( )

∞ =

, = < +∞ = ,

∑

i k k

F i j P T F i j ( )

1 1 1

{ } ( ) 1 ( )

j j m m m

m P N m mF j j F j j

= ∞ − =

= , − ,

∑ ∑

1

1 1

m ∞

∑

♣ ( ) , F i j expresses the probability: starting at i the MC will ever visit state j . ( ) ( ) ( ) ( ) , = , + , , , ∈

∑

F i j P i j P i b F b j i E ( ) ( ) ( )

2

1 1 1 ( ) 1 ( ) 1 ( ) F j j F j j F j j = − , = − , − ,

1 2 1

, 1 (1 )

m m

m x x x

− =

= < −

∑

{ } ∈ −

∑

b E j

If by

j

N we denote the total number of visits to state j , then { } = =

j j

P N m

( )

1

( ) 1 ( )

−

, − ,

m

F j j F j j 1 ( ) − , = ⎧ ⎨ F i j m

• If

( ) 1 , = ⇒ = +∞

j

F j j N w.p.1. Therefore, if [ ] = ⇒ = +∞

j j

X j E N

• If

( ) 1 , < F j j then

j

N follows geometric distribution with probability of success

and for ≠

i j , { } = =

i j

P N m

( )

1

( ) ( ) ( ) 1 ( ) 1 2

−

, ⎧ ⎨ , , − , = , , ⎩

m

j F i j F j j F j j m … ( ) , j j

j

g p y 1 ( ) = − , p F j j . Hence,

1 1 1 ( )

[ ]

− ,

= =

j j p F j j

E N >From the previous we obtain the Corollary: 1 ( ) 1 { } ( ) 1 , < ⎧ < +∞ = ⎨ = ⎩

j j

F j j P N F j j

28

( ) 1 , = ⎩ F j j

ΠΜΣ524: Μοντελοποίηση και Ανάλυση Απόδοσης Δικτύων (Ι. Σταυρακάκης / Α. Παναγάκης - ΕΚΠΑ)

SLIDE 15

Let ( ) [ ] , =

i j

R i j E N ( R is called the potential matrix of X ) Then,

( ) ( ) ( ) (1 ( )) 0 R i j F i j R j j F i j

( ) , = R j j

1 1 ( ) − , F j j

( ) , = R i j ( , ) ( ) F i j R j j ,

, (i≠j)

( , ) ( , ) ( , ) (1 ( , )) 0 R i j F i j R j j F i j = + −

Computation of ( ) , R i j first and then ( ) , F i j Define: 1 ( ) 1 1 ( ) 1 ( ( )) ( ) ω ω ω , = , = ⎧ ⎧ = ⇒ = ⎨ ⎨ , ≠ , ≠ ⎩ ⎩

n j j n n

X j k j k X X j k j Then, ( ) ω =

j

N

01 (

( )) ω

∞ =

∑

j n n

X

∞ ∞ ∞ ∞

⎡ ⎤ ⎡ ⎤

∑ ∑ ∑ ∑

( ) , = R i j 1 ( ) 1 ( ) { } ( )

∞ ∞ ∞ ∞ = = = =

⎡ ⎤ ⎡ ⎤ = = = = , ⎣ ⎦ ⎣ ⎦

∑ ∑ ∑ ∑

n i j n i j n i n n n n n

E X E X P X j P i j In matrix notation:

2 2

= + + + ⇒ = = + + = − L L R I P P RP PR P P R I from which we obtain ( ) ( ) R I P I P R I

ΠΜΣ524: Μοντελοποίηση και Ανάλυση Απόδοσης Δικτύων (Ι. Σταυρακάκης / Α. Παναγάκης - ΕΚΠΑ) 29

( ) ( ) − = − = R I P I P R I

Classification of states

X: MC, with state space E , transition matrix P T : The time of first visit to state j

j

N : The total number of visits to state j

j

j Definition ♣ State j is called recurrent if { } 1 < ∞ =

j

P T ♣ State j is called recurrent if { } 1 <

j

P T ♣ State j is called transient if { } = ∞ >

j

P T ♣ A recurrent state j is called null if [ ] = ∞

j

E T A j i ll d ll if [ ] E T ♣ A recurrent state j is called non-null if [ ] < ∞

j

E T ♣ A recurrent state j is called periodic with period δ , if 2 δ ≥ is the greatest integer for which { for some 1} 1 δ = ≥ =

j

P T n n

ΠΜΣ524: Μοντελοποίηση και Ανάλυση Απόδοσης Δικτύων (Ι. Σταυρακάκης / Α. Παναγάκης - ΕΚΠΑ) 30

SLIDE 16

• If j is recurrent then starting at j the probability of returning to j is 1.

If j is recurrent then starting at j the probability of returning to j is 1. ( ) 1 ( ) [ ] { } 1

j j j j

F j j R j j E N P N , = ⇒ , = = +∞ ⇐⇒ = +∞ =

• If j is transient then there exists a positive probability 1

( ) − , F j j

• f never

returning to j . ( ) 1 ( ) [ ] { } 1 , < ⇒ , = < ∞ ⇐⇒ < ∞ =

j j j j

F j j R j j E N P N I thi ( ) ( ) ( ) ( ) < < R i j F i j R j j R j j d i ( ) ( )

∑

n

R i j P i j In this case ( ) ( ) ( ) ( ) , = , , < , < ∞ R i j F i j R j j R j j and since ( ) ( ) , = ,

∑

n n

R i j P i j we conclude that lim ( ) , →

n

P i j lim ( )

→∞

, →

n

P i j

ΠΜΣ524: Μοντελοποίηση και Ανάλυση Απόδοσης Δικτύων (Ι. Σταυρακάκης / Α. Παναγάκης - ΕΚΠΑ) 31

Theorem: ♣ If j transient or recurrent null then lim ( ) ∀ ∈ →

n

i E P i j lim ( )

→∞

∀ ∈ , , →

n

i E P i j ♣ If j recurrent non-null then ( ) lim ( ) and lim ( ) ( ) ( ) π π

→∞ →∞

= , > ∀ ∈ , , = ,

n n n n

j P j j i E P i j F i j j ♣ If j periodic with period δ , then a retutrn to j is possible only at steps numbered δ , 2δ , 3δ , ... ( ) { } 0 only if {0 2 } δ δ , = = > ∈ , , ,

n j n

P j j P X j n …

ΠΜΣ524: Μοντελοποίηση και Ανάλυση Απόδοσης Δικτύων (Ι. Σταυρακάκης / Α. Παναγάκης - ΕΚΠΑ) 32

SLIDE 17

R t ll R t ll T i t Recurrent non-null Recurrent null Transient { } 1 < ∞ =

j

P T { } = ∞ >

j

P T [ ] < ∞

j

E T [ ] = ∞

j

E T ( ) ( ) [ ] ( ) 1 ( ) [ ] { } 1

j j j j

F j j R j j E N P N , = ⇒ , = = +∞ ⇐⇒ = +∞ = ( ) 1 ( ) [ ] { } 1

j j j j

F j j R j j E N P N , < ⇒ , = < ∞ ⇐⇒ < ∞ = ( ) lim ( ) and

n

j P j j i E π = , > ∀ ∈ , li ( )

n

lim ( ) ( ) ( )

n n n

P i j F i j j π

→∞ →∞

, = , lim ( )

n n

i E P i j

→∞

∀ ∈ , , → ♣ A recurrent state j is called periodic with periodδ , if 2 δ ≥ is the greatest integer for which { for some 1} 1 δ = ≥ =

j

P T n n

j

♣ If j periodic with period δ , then a return to j is possible only at steps numbered δ , 2δ , 3δ , ... ( ) { } 0 only if {0 2 } δ δ = = > ∈

n

P j j P X j n ( ) { } 0 only if {0 2 } δ δ , = = > ∈ , , ,

j n

P j j P X j n …

33 ΠΜΣ524: Μοντελοποίηση και Ανάλυση Απόδοσης Δικτύων (Ι. Σταυρακάκης / Α. Παναγάκης - ΕΚΠΑ)

We say that state j can be reached from state i → , i j if ( )

n

n P i j ∃ ≥ : , > → , i j iff ( ) F i j , > Definition:

• A set of states is closed if no state outside it can be reached from any state in it

A set of states is closed if no state outside it can be reached from any state in it.

• A state forming a closed set by itself is called an absorbing state
• A closed set is called irreducible if no proper subset of it is closed.
• A MC is called irreducible if its only closed set is the set of all states

A MC is called irreducible if its only closed set is the set of all states Comments: ♣ If j is absorbing then ( ) 1 , = P j j . ♣ If MC is irreducible then all states can be reached from each other. ♣ If

1 2

{ } = , , ∈ L C c c E is a closed set and ( ) ( ) , = ,

i j

Q i j P c c , , ∈

i j

c c C , then Q is a Markov matrix. ♣ If → i j and → j k then → i k . To find the closed set C that contains i we work as follows:

• Starting with i we include in C all states j that can be reached from i :

( ) , > P i j .

• We next include in C all states k that can be reached from j :

( ) , > P j k .

• We repeat the previous step

ΠΜΣ524: Μοντελοποίηση και Ανάλυση Απόδοσης Δικτύων (Ι. Σταυρακάκης / Α. Παναγάκης - ΕΚΠΑ) 34

• We repeat the previous step

SLIDE 18

Example: MC with state space { } = E a b c d e and transition matrix Example: MC with state space { } = , , , , E a b c d e and transition matrix 1 1 2 2 ⎛ ⎞ ⎜ ⎟ ⎜ ⎟ Comments:

• Closed sets: {

} , , a c e and { } , , , , a b c d e 1 3 4 4 1 2 ⎜ ⎟ ⎜ ⎟ ⎜ ⎟ ⎜ ⎟ ⎜ ⎟ P Closed sets: { } , , a c e and { } , , , , a b c d e

• There are two closed sets. Thus, the MC is not

irreducible. 3 3 1 1 1 4 2 4 ⎜ ⎟ = ⎜ ⎟ ⎜ ⎟ ⎜ ⎟ ⎜ ⎟ P 4 2 4 1 1 1 3 3 3 ⎜ ⎟ ⎜ ⎟ ⎜ ⎟ ⎝ ⎠

c e

ΠΜΣ524: Μοντελοποίηση και Ανάλυση Απόδοσης Δικτύων (Ι. Σταυρακάκης / Α. Παναγάκης - ΕΚΠΑ) 35

Example: MC with state space { } = E a b c d e and transition matrix Example: MC with state space { } , , , , E a b c d e and transition matrix 1 1 2 2 1 3 ⎛ ⎞ ⎜ ⎟ ⎜ ⎟ ⎜ ⎟ Comments:

• Closed sets: {

} , , a c e and { } , , , , a b c d e

• There are two closed sets. Thus, the MC is not

1 3 4 4 1 2 3 3 ⎜ ⎟ ⎜ ⎟ ⎜ ⎟ ⎜ ⎟ = ⎜ ⎟ P irreducible.

• If we delete the 2nd and 4th rows we obtain the

Markov matrix: ⎛ ⎞ 3 3 1 1 1 4 2 4 1 1 1 ⎜ ⎟ ⎜ ⎟ ⎜ ⎟ ⎜ ⎟ ⎜ ⎟ ⎜ ⎟ 1 1 2 2 1 2 3 3 ⎛ ⎞ ⎜ ⎟ ⎜ ⎟ ⎜ ⎟ = ⎜ ⎟ Q

1 1 ⎛ ⎞

1 1 1 3 3 3 ⎜ ⎟ ⎝ ⎠ 3 3 1 1 1 3 3 3 ⎜ ⎟ ⎜ ⎟ ⎜ ⎟ ⎜ ⎟ ⎝ ⎠ Q

1 1 2 2 1 2 ⎛ ⎞ ⎜ ⎟ ⎜ ⎟ ⎜ ⎟ ⎜ ⎟ If we relabel the states 1 = a , 2 = c , 3 = e, 4 = b and 5 = d we get 3 3 1 1 1 3 3 3 ⎜ ⎟ ⎜ ⎟ ⎜ ⎟ = ⎜ ⎟ ⎜ ⎟ P 1 3 4 4 1 1 1 ⎜ ⎟ ⎜ ⎟ ⎜ ⎟ ⎜ ⎟ ⎜ ⎟

ΠΜΣ524: Μοντελοποίηση και Ανάλυση Απόδοσης Δικτύων (Ι. Σταυρακάκης / Α. Παναγάκης - ΕΚΠΑ) 36

4 2 4 ⎜ ⎟ ⎝ ⎠

SLIDE 19

Lemma If j recurrent and → ⇒ → j k k j . Thus, ( ) 1 , = F k j . Proof: If → j k then k is reached without returning to j with probability a. Once k j g j p y is reached, the probability that j is never visited again is 1 ( ) − , F k j . Hence, 1 ( ) a(1 ( )) − , ≥ − , ≥ F j j F k j But j is recurrent, so that ( ) 1 ( ) 1 , = ⇒ , = F j j F k j But j is recurrent, so that ( ) 1 ( ) 1 , ⇒ , F j j F k j ♠ As a result: If → j k but → / k j , then j must be transient. Theorem: From recurrent states only recurrent states can be reached. Theorem: In a Marcov chain the recurrent states can be divided in a unique manner, into irreducible closed sets

1

C ,

2

C , …, and after an appropriate arrangement:

1, 2 ,

, pp p g

1 2

⎛ ⎞ ⎜ ⎟ ⎜ ⎟ ⎜ ⎟ L L P P

3 1 2 3

⎜ ⎟ = ⎜ ⎟ ⎜ ⎟ ⎜ ⎟ ⎝ ⎠ L L L L O M L P P Q Q Q Q

ΠΜΣ524: Μοντελοποίηση και Ανάλυση Απόδοσης Δικτύων (Ι. Σταυρακάκης / Α. Παναγάκης - ΕΚΠΑ) 37

1 2 3

⎝ ⎠ Q Q Q Q Theorem: Let X an irreducible MC. Then, one of the following holds:

• All states are transient
• All states are transient.
• All states are recurrent null
• All states are recurrent non-null
• Either all aperiodic or if one is periodic with period δ , all are periodic

Either all aperiodic or if one is periodic with period δ , all are periodic with the same period. Proof: Since X is irreducible then → j k and → k j , which means that ∃ , : r s ( ) , >

r

P j k and ( ) , >

s

P k j . Pick the smallest , r s and let ( ) ( ) β = , ,

r s

P j k P k j .

• If k recurrent ⇒ j recurrent.
• If k transient ⇒ j transient (If it was recurrent then k would be recurrent)
• If k transient ⇒ j transient. (If it was recurrent then k would be recurrent)
• If k recurrent null then

( ) , →

m

P k k as → ∞ m . But ( ) ( ) ( ) β

+ +

, ≥ , ⇒ , →

n r s n n

P k k P j j P j j

ΠΜΣ524: Μοντελοποίηση και Ανάλυση Απόδοσης Δικτύων (Ι. Σταυρακάκης / Α. Παναγάκης - ΕΚΠΑ) 38

SLIDE 20

C f C i d ibl l d f fi i h / ll Corollary: If C irreducible closed set of finitely many states, then ∃ / recurrent null states. Proof: If one is recurrent null then all states are recurrent null Proof: If one is recurrent null then all states are recurrent null. Thus, lim ( )

→∞

, = , ∀ , ∈

n n

P i j i j C . But, ( ) 1 lim ( ) 1

→∞

∀ ∈ , ≥ , , = ⇒ , =

∑ ∑

n n n

i C n P i j P i j

→∞ ∈ ∈ n j C j C

Because, we have finite number of states lim ( ) lim ( )

→∞ →∞ ∈ ∈

, = , =

∑ ∑

n n n n j C j C

P i j P i j

∈ ∈ j C j C

Corollary: If C is an irreducible closed set with finitely many states then there are no transient states

ΠΜΣ524: Μοντελοποίηση και Ανάλυση Απόδοσης Δικτύων (Ι. Σταυρακάκης / Α. Παναγάκης - ΕΚΠΑ) 39

Algorithm - Finite number of states

• Identify irreducible closed sets.
• All states belonging to an irreducible closed set are recurrent positive
• The rest of the states are transient
• Periodicity is checked to each irreducible set

ΠΜΣ524: Μοντελοποίηση και Ανάλυση Απόδοσης Δικτύων (Ι. Σταυρακάκης / Α. Παναγάκης - ΕΚΠΑ) 40

SLIDE 21

Example: The irreducible closed sets are {1 3} , , {2 7 9} , , and {6}. The states {4 5 8 10} , , , are t i t If l b l th t t bt i

• transient. If we relabel the states we obtain

1 1 1 ⎛ ⎞ ⎜ ⎟ ⎜ ⎟ 2 2 1 1 2 ⎜ ⎟ ⎜ ⎟ ⎜ ⎟ ⎜ ⎟ ⎜ ⎟ 3 3 1 3 4 4 ⎜ ⎟ ⎜ ⎟ ⎜ ⎟ ⎜ ⎟ ⎜ ⎟ 4 4 1 1 ⎜ ⎟ = ⎜ ⎟ ⎜ ⎟ ⎜ ⎟ ⎜ ⎟ P 1 1 1 3 3 3 1 1 1 1 ⎜ ⎟ ⎜ ⎟ ⎜ ⎟ ⎜ ⎟ ⎜ ⎟ 4 4 4 4 1 1 1 3 3 3 ⎜ ⎟ ⎜ ⎟ ⎜ ⎟ ⎜ ⎟ ⎝ ⎠

ΠΜΣ524: Μοντελοποίηση και Ανάλυση Απόδοσης Δικτύων (Ι. Σταυρακάκης / Α. Παναγάκης - ΕΚΠΑ) 41

3 3 3 ⎝ ⎠ Example: Let

n

N the number of successes in the first n Bernoulli trials. As we have seen 1 = + ⎧p j i

1

1 ( ) { }

• therwise

+

+ ⎧ ⎪ , = = | = = = ⎨ ⎪ ⎩

n n

p j i P i j P N j N i q j i Thus, ⎛ ⎞ ⎜ ⎟ L q p ⎜ ⎟ ⎜ ⎟ = ⎜ ⎟ ⎜ ⎟ ⎝ ⎠ L L M M M O q p P q ⎝ ⎠ M M M O ∀j we have 1 → + j j but 1 + → / j j . This means that j is not recurrent. Since the MC is irreducible all states are transient MC is irreducible all states are transient.

ΠΜΣ524: Μοντελοποίηση και Ανάλυση Απόδοσης Δικτύων (Ι. Σταυρακάκης / Α. Παναγάκης - ΕΚΠΑ) 42

SLIDE 22

Example: Remaining lifetime Remember:

1 1

( ) 1 ( ) 1 ( ) ( ) 1 ( ) ω ω ω ω ω

+

− ≥ ⎧ = ⎨ − = ⎩

n n n

X X X Z X

1( )

1 ( ) ω ω

+

⎩

n n

Z X from which we obtain: 1 ≥ i

1

1 1 ( ) { } { 1 } 1

n n n n

j i P i j P X j X j P X j X j j i

+

= − ⎧ , = = | = = − = | = = ⎨ ⎩

1

( ) { } { } 1

n n n n

j j j j j j i

+

| | ⎨ ≠ − ⎩ = i

1 1

(0 ) { 0} { 1 0}

+ +

, = = | = = − = | =

n n n n

P j P X j X P Z j X

1 1

{ 1}

+ +

= = + =

n j

P Z j p

1 1

{ }

+ + n j

j p

1 2 3

⎛ ⎞ ⎜ ⎟ L p p p

1 2 3

1 1 1 ⎜ ⎟ ⎜ ⎟ ⎜ ⎟ = ⎜ ⎟ ⎜ ⎟ L L P 1 ⎜ ⎟ ⎜ ⎟ ⎝ ⎠ L M M M O

ΠΜΣ524: Μοντελοποίηση και Ανάλυση Απόδοσης Δικτύων (Ι. Σταυρακάκης / Α. Παναγάκης - ΕΚΠΑ) 43

1 2 3

1 ⎛ ⎞ ⎜ ⎟ ⎜ ⎟ L L p p p 1 1 1 ⎜ ⎟ ⎜ ⎟ = ⎜ ⎟ ⎜ ⎟ ⎜ ⎟ L L L P ⎜ ⎟ ⎝ ⎠ M M M O >From state 0 we reach state j in one step. From j we can reach 1 − j , 2 − j , ..., 1, 0 . Thus, all states can be reached from each other, which means that the MC is 0 . Thus, all states can be reached from each other, which means that the MC is

• irreducible. Since,

(0 0) , > P the MC is aperiodic. Return to state 0 occurs if the lifetime is finite: 1 (0 0) 1 = ⇒ = =

∑ ∑

p F p 1 (0 0) 1 = ⇒ , = =

∑ ∑

j j j j

p F p Since state 0 is recurrent, all states are recurrent. If the expected lifetime: = +∞

∑

j j

jp then state 0 is null and all states are recurrent null. If h d lif i If the expected lifetime: < ∞

∑

j j

jp then state 0 is non null and all states are recurrent non null

ΠΜΣ524: Μοντελοποίηση και Ανάλυση Απόδοσης Δικτύων (Ι. Σταυρακάκης / Α. Παναγάκης - ΕΚΠΑ) 44

then state 0 is non-null and all states are recurrent non-null.

SLIDE 23

Algorithm - Infinite number of states Algorithm - Infinite number of states Theorem: Let X an irreducible MC, and consider the system of linear equations: ( ) ( ) ( ) ν ν = , , ∈

∑

j i P i j j E ( ) ( ) ( ) ν ν

∈

, , ∈

∑

i E

j i P i j j E Then all states are recurrent non-null iff there exists a solution ν with ( ) 1 ν =

∑

j

∈

∑

j E

Theorem: Let X an irreducible MC with transition matrix P , and let Q be the i b i d f P b d l i h k d k l f k E Th matrix obtained from P by deleting the k -row and k -column for some ∈ k E . Then all states are recurrent if and only if the only solution of ( ) ( ) ( ) ( ) 1

∈

= , , ≤ ≤ , ∈

∑

j E

h i Q i j h j h i i E

∈ j E

is ( ) = h i for all ∈ i E . { } = − E E k .

• Use first theorem to determine whether all states are recurrent non-null or not.

I th l tt th d th t d t i h th th t t

• In the latter case, use the second theorem to determine whether the states are

transient or not.

ΠΜΣ524: Μοντελοποίηση και Ανάλυση Απόδοσης Δικτύων (Ι. Σταυρακάκης / Α. Παναγάκης - ΕΚΠΑ) 45

Example: Random walks.

1 ⎛ ⎞ L ⎛ ⎞ ⎜ ⎟ ⎜ ⎟ =⎜ ⎟ ⎜ ⎟ L L q p P q p ⎜ ⎟ ⎝ ⎠ M M M M O

• All states can be reached from each other and thus the chain is irreducible
• All states can be reached from each other, and thus the chain is irreducible.
• A return to state 0 can occur only at steps numbered 2,4,6,... Therefore, state 0 is

periodic with period

2 δ = .

• Since X is irreducible all states are periodic with period 2
• Since X is irreducible all states are periodic with period 2.
• Either all states are recurrent null, or all are recurrent non-null, or all the states are

transient. Check for a solution of ν

νP

Check for a solution of ν

ν = P. ν =

1

ν q

1

ν =

2

ν ν +q

1 2

q

2

ν =

1 3

ν ν + p q

3

ν =

2 4

ν ν + p q

46 ΠΜΣ524: Μοντελοποίηση και Ανάλυση Απόδοσης Δικτύων (Ι. Σταυρακάκης / Α. Παναγάκης - ΕΚΠΑ)

SLIDE 24

Hence,

1

ν = 1 ν

q q 2

ν =

2

1 1

ν ν ν

⎛ ⎞ ⎜ ⎟ ⎝ ⎠

− =

p q q q 3

ν = (

)

2 2 3

1

ν ν − =

p p p q q 3

( )

2 3

q q q q

Any solution is of the form

1

1 1 2 ν ν

−

⎛ ⎞ = = ⎜ ⎟

j

p j 1 2 ν ν = , = , , ⎜ ⎟ ⎝ ⎠

j

j … q q

If

< p q , then 1 / < p q

and

1

1 2

− ∞ ∞

⎛ ⎞ ⎛ ⎞

j 1

1 2 1 ν ν ν

∞ ∞ = =

⎛ ⎞ ⎛ ⎞ ⎜ ⎟ = + = ⎜ ⎟ ⎜ ⎟ − ⎝ ⎠ ⎝ ⎠

∑ ∑

j j j

p q q q q p If we choose ν

− q p then

1 ν

∑

and If we choose

2

ν =

q p q then

1 ν =

∑

j

and 1 1 2 ⎧ ⎛ ⎞ − , = ⎪ ⎜ ⎟ ⎝ ⎠ ⎪ p j q

1

2 ( ) 1 1 1 2 ν

−

⎝ ⎠ ⎪ = ⎨ ⎛ ⎞⎛ ⎞ ⎪ − , ≥ ⎜ ⎟⎜ ⎟ ⎪ ⎝ ⎠⎝ ⎠ ⎩

j

q j p p j q q q

47

⎝ ⎠⎝ ⎠ ⎩ In this case all states are recurrent non null

ΠΜΣ524: Μοντελοποίηση και Ανάλυση Απόδοσης Δικτύων (Ι. Σταυρακάκης / Α. Παναγάκης - ΕΚΠΑ)

If > p q either all states are recurrent null or all states are transient. Consider the matri matrix ⎛ ⎞ ⎜ ⎟ ⎜ ⎟ = ⎜ ⎟ L L p q p Q ⎜ ⎟ ⎜ ⎟ ⎝ ⎠ L M M M M O Q q p The equation = h Qh gives ( ( ) =

i

h h i )

1 1 1

1

− +

⎡ ⎤ ⎛ ⎞ ⎛ ⎞ = + + + + ⎢ ⎥ ⎜ ⎟ ⎜ ⎟ ⎢ ⎥ ⎝ ⎠ ⎝ ⎠ ⎣ ⎦ L

i i i

q q q h h p p p

• If

= p q then

1

=

i

h ih for all 1 ≥ i and the only way to have 0 1 ≤ ≤

i

h for all i is by choosing

1

= h which implies =

i

h that is all states are recurrent null.

• If

> p q , then choosing

1

1 ( ) = − / h q p , we get 1 ⎛ ⎞ = − ⎜ ⎟ ⎝ ⎠

i i

q h p ⎝ ⎠ p which also satisfies 0 1 ≤ ≤

i

h . In this case all states are transient.

48 ΠΜΣ524: Μοντελοποίηση και Ανάλυση Απόδοσης Δικτύων (Ι. Σταυρακάκης / Α. Παναγάκης - ΕΚΠΑ)

SLIDE 25

Calculation of R and F ( ) [ ] d b f i i ♣ ( ) [ ]

i j

R i j E N , = Expected number of visits to state j . ♣ ( ) F i j , = The probability of ever reaching state j starting at i . j Recurrent state: ( ) 1 ( ) F j j R j j , = ⇒ , = ∞ ( ) R i j = ( ) ( ) F i j R j j ( ) R i j = ( ) F i j , = ⎧ ⎨ ( ) R i j , = ( ) ( ) F i j R j j , , ( ) R i j , = ( ) F i j ⎨+∞ , > ⎩ j Transient / i Recurrent state: ( ) ( ) F i j R i j , = ⇒ , = i j , Transient Let { the transient states } D = , ( ) ( ) Q i j P i j , = , , ( ) ( ) S i j R i j , = , , i j D , ∈ . Then

m m m

K K P P L Q L Q

⎛ ⎞ ⎜ ⎟ ⎜ ⎟ ⎜ ⎟ ⎜ ⎟ ⎝ ⎠

⎛ ⎞ = ⇒ = ⎜ ⎟ ⎝ ⎠

m

L Q L Q

⎜ ⎟ ⎝ ⎠

⎝ ⎠ Hence

2 m m m

K R P S Q I Q Q

⎛ ⎞ ∞ ∞ ⎜ ⎟ ⎜ ⎟ ⎜ ⎟

= = ⇒ = = + + +

∑ ∑ ∑

L

ΠΜΣ524: Μοντελοποίηση και Ανάλυση Απόδοσης Δικτύων (Ι. Σταυρακάκης / Α. Παναγάκης - ΕΚΠΑ) 49

Hence,

m m m m

R P S Q I Q Q L Q

⎜ ⎟ ⎜ ⎟ = = ⎜ ⎟ ⎝ ⎠

⇒ + + +

∑ ∑ ∑ ∑

Computation of S S =

2

I Q Q + + + ⇒ L SQ QS = =

2

Q Q S I + + = − ⇒ L SQ QS Q Q S I + + ⇒ ( ) I Q S I − =

, (

) S I Q I − = Proposition: If there are finitely many transient states

1

( ) S I Q − = − Proposition: If there are finitely many transient states ( ) S I Q = − ♣ When the set D of transient states is infinite, it is possible to have more than one solution to the system. solution to the system. Theorem: S is the minimal solution of ( ) I Q Y I − = , Y ≥ Theorem: S is the unique solution of ( ) I Q Y I − = if and only if the only bounded solution of h Qh = is h = , or equivalently 1 h Qh h h = , ≤ ≤ ⇐⇒ =

ΠΜΣ524: Μοντελοποίηση και Ανάλυση Απόδοσης Δικτύων (Ι. Σταυρακάκης / Α. Παναγάκης - ΕΚΠΑ) 50

SLIDE 26

Example: Let X a MC with state space {1 2 3 4 5 6 7 8} E = , , , , , , , 0 4 0 3 0 3 0 6 0 4 0 5 0 5 . . . | | ⎛ ⎞ ⎜ ⎟ . . . | | ⎜ ⎟ ⎜ ⎟ | | {1 2 3} , , are recurrent positive aperiodic. 0 5 0 5 1 P ⎜ ⎟ . . . | | ⎜ ⎟ − − − | − − | − − − ⎜ ⎟ ⎜ ⎟ | . . | = ⎜ ⎟ p {4 5} , are recurrent positive aperiodic. 0 8 0 2 0 4 0 6 P ⎜ ⎟ | . . | ⎜ ⎟ ⎜ ⎟ − − − | − − | − − − ⎜ ⎟ . . . | | . . . ⎜ ⎟ ⎜ ⎟ {6 7 8} , , are transient 0 4 0 4 0 2 0 1 0 3 0 6 ⎜ ⎟ . . . | | . . . ⎜ ⎟ ⎜ ⎟ . . . | | . . . ⎝ ⎠

1

0 4 0 6 0 6 0 6

−

. . . . − . . ⎛ ⎞ ⎛ ⎞ ⎜ ⎟ ⎜ ⎟

1

0 2 ( ) 1 0 2 0 6 0 6 1 Q S I Q − ⎜ ⎟ ⎜ ⎟ = . . . ⇒ = − = . . − . ⎜ ⎟ ⎜ ⎟ ⎜ ⎟ ⎜ ⎟ . . . − . . . ⎝ ⎠ ⎝ ⎠

ΠΜΣ524: Μοντελοποίηση και Ανάλυση Απόδοσης Δικτύων (Ι. Σταυρακάκης / Α. Παναγάκης - ΕΚΠΑ) 51

recurrent, can be reached from j i transient, recurrent j i recurrent cannot be reached from j i transient j i transient j recurrent j recurrent, cannot be reached from j i , transient j i

∞ ∞ ∞ | | ⎛ ⎞ ⎜ ⎟ ∞ ∞ ∞ | | ⎜ ⎟ ⎜ ⎟ ⎜ ⎟ ∞ ∞ ∞ | | ⎜ ⎟ − − − | − − | − − − ⎜ ⎟ ⎜ ⎟ | ∞ ∞ | ⎜ ⎟

recurrent i

125 75 15 R | | ⎜ ⎟ | ∞ ∞ | ⎜ ⎟ = ⎜ ⎟ − − − | − − | − − − ⎜ ⎟ ⎜ ⎟ 125 75 15 66 66 66 15 75 15 ⎜ ⎟ ∞ ∞ ∞ | | ⎜ ⎟ ⎜ ⎟ ⎜ ⎟ ∞ ∞ ∞ | | ⎜ ⎟ S

transient i

66 66 66 75 45 75 66 66 66 ∞ ∞ ∞ | | ⎜ ⎟ ⎜ ⎟ ⎜ ⎟ ∞ ∞ ∞ | | ⎜ ⎟ ⎝ ⎠

t a s e t i

ΠΜΣ524: Μοντελοποίηση και Ανάλυση Απόδοσης Δικτύων (Ι. Σταυρακάκης / Α. Παναγάκης - ΕΚΠΑ) 52

⎝ ⎠

SLIDE 27

Computation of ( ) F i j , ♣ i j , recurrent belonging to the same irreducible closed set j , g g ( ) 1 F i j , = ♣ i j , recurrent belonging to different irreducible closed sets j , g g ( ) F i j , = ♣ i j , transient Then ( ) R i j , < ∞ and j , ( ) j , 1 ( ) ( ) 1 ( ) ( ) ( ) R i j F j j F i j R j j R j j , , = − , , = , , ♣ i transient, j recurrent ???? Lemma: If C is irreducible closed set of recurrent states then for any transient state i : Lemma: If C is irreducible closed set of recurrent states, then for any transient state i : ( ) ( ) F i j F i k , = , for all j , k C ∈ . Proof: For ( ) ( ) 1 j k C F j k F k j , ∈ ⇒ , = , = . Thus, once the chain reaches any one of the states of C , it also visits all the other states. Hence, ( ) ( ) F i j F i k , = , is the probability of entering the set C from i

ΠΜΣ524: Μοντελοποίηση και Ανάλυση Απόδοσης Δικτύων (Ι. Σταυρακάκης / Α. Παναγάκης - ΕΚΠΑ) 53

entering the set C from i .

Let Lump all states of

j

C together to make one absorbing state:

1 2

P P ⎛ ⎞ ⎜ ⎟ ⎜ ⎟ p

j

g g 1 1 1 ⎛ ⎞ ⎜ ⎟ ⎜ ⎟ ⎜ ⎟

3 1 2 3

P P Q Q Q Q ⎜ ⎟ ⎜ ⎟ = ⎜ ⎟ ⎜ ⎟ ⎜ ⎟ ⎝ ⎠ O 1 ˆ ( ) ( ) 1

j

j k C

P b i P i k i D

∈

⎜ ⎟ = , = , , ∈ ⎜ ⎟ ⎜ ⎟ ⎜ ⎟ ⎜ ⎟ ⎜ ⎟

∑

O

1 2 3

Q Q Q Q ⎝ ⎠

1 2 3 m

b b b b Q ⎜ ⎟ ⎜ ⎟ ⎝ ⎠ L Th b bili f hi h b bi j f h i i b h The probability of ever reaching the absorbing state j from the transient state i by the chain with the transition matrix ˆ P is the same as that of ever reaching

j

C from i . ˆ ( ) ( ) I P B b b B i j P i k ı D

⎡ ⎤

⎛ ⎞ ∈ ⎜ ⎟

∑

1

( ) ( )

j

m k C

P B b b B i j P i k ı D B Q

⎡ ⎤ ⎢ ⎥ ⎣ ⎦ ∈

= , = , , = , , ∈ ⎜ ⎟ ⎝ ⎠

∑

L

2 1

ˆ

n n

I

⎛ ⎞ ⎜ ⎟ 2 1

ˆ ( )

n n n n n

B I Q Q Q B P B Q

⎜ ⎟ − ⎜ ⎟ ⎜ ⎟ ⎜ ⎟ ⎝ ⎠

= , = + + + + L ( ) B i j , is the probability that starting from i , the chain enters the recurrent class

j

C

ΠΜΣ524: Μοντελοποίηση και Ανάλυση Απόδοσης Δικτύων (Ι. Σταυρακάκης / Α. Παναγάκης - ΕΚΠΑ) 54

( )

n

B i j , is the probability that starting from i , the chain enters the recurrent class

j

C

SLIDE 28

Define: lim

k n n k

G B Q B SB

∞ ⎛ ⎞ ⎜ ⎟ ⎜ ⎟ ⎜ ⎟ →∞ = ⎝ ⎠

= = =

∑

♣ ( ) G i j , is the probability of ever reaching the set

j

C from the transient state i : ( ( ) F i j , ) Proposition: Let Q the matrix obtained from P by deleting all the rows and columns corresponding to the recurrent states, and let B be defined as previously, for each t i t i d t l C transient i and recurrent class

j

C .

• Compute S
• Compute G

SB =

• (

) ( ) G i j F i k , = , ,

j

k C ∀ ∈ .

• If there is only one recurrent class and finitely many transient states, then things are

different. In this case, it can be proved that: 1 ( ) 1 G F i j j C = ⇒ , = , ∀ ∈

ΠΜΣ524: Μοντελοποίηση και Ανάλυση Απόδοσης Δικτύων (Ι. Σταυρακάκης / Α. Παναγάκης - ΕΚΠΑ) 55

Example: Let X a MC with state space {1 2 3 4 5 6 7 8} E = , , , , , , , ⎛ ⎞

0 4 0 3 0 3 0 6 0 4 0 5 0 5 . . . | | . . . | | . . . | |

⎛ ⎞ ⎜ ⎟ ⎜ ⎟ ⎜ ⎟ ⎜ ⎟

1 0 8 0 2 P | | − − − | − − | − − − | . . | = | |

⎜ ⎟ ⎜ ⎟ ⎜ ⎟ ⎜ ⎟ ⎜ ⎟

0 8 0 2 0 4 0 6 | . . | − − − | − − | − − − . . . | | . . .

⎜ ⎟ ⎜ ⎟ ⎜ ⎟ ⎜ ⎟ ⎜ ⎟

0 4 0 4 0 2 0 1 0 3 0 6 . . . | | . . . . . . | | . . .

⎜ ⎟ ⎜ ⎟ ⎜ ⎟ ⎜ ⎟ ⎝ ⎠

ΠΜΣ524: Μοντελοποίηση και Ανάλυση Απόδοσης Δικτύων (Ι. Σταυρακάκης / Α. Παναγάκης - ΕΚΠΑ) 56

SLIDE 29

, recurrent belonging to the same irreducible closed set i j recurrent belonging to different irreducible closed sets i j

⎛ ⎞

transient j recurrent j , recurrent belonging to different irreducible closed sets i j

1 1 1 1 1 1 1 1 1 | | ⎛ ⎞ ⎜ ⎟ | | ⎜ ⎟ ⎜ ⎟ | | ⎜ ⎟

transient, recurrent j i

1 1 1 1 1 F ⎜ ⎟ | | ⎜ ⎟ − − − | − − | − − − ⎜ ⎟ ⎜ ⎟ | | = ⎜ ⎟

recurrent i

1 1 1 1 1 0 472 1 0 20 ⎜ ⎟ | | ⎜ ⎟ ⎜ ⎟ − − − | − − | − − − ⎜ ⎟ | | . . . ⎜ ⎟

1

, transient j i

1 1 1 0 12 0 12 0 20 1 1 1 0 60 0 60 0 12 | | ⎜ ⎟ ⎜ ⎟ | | . . . ⎜ ⎟ ⎜ ⎟ | | . . . ⎝ ⎠

transient i

1 ( ) 1 ( ) ( ) ( ) F j j R j j R i j F i j , = − , , , = ( ) ( ) F i j R j j , = ,

( h bl ) t l d fi it l t i t t t

ΠΜΣ524: Μοντελοποίηση και Ανάλυση Απόδοσης Δικτύων (Ι. Σταυρακάκης / Α. Παναγάκης - ΕΚΠΑ) 57

• ne (reachable) recurrent class and finitely many transient states

Example: 0 5 0 5 . . ⎛ ⎞ ⎜ ⎟ 0 8 0 2 1 0 4 0 6 1 ˆ 1 P P ⎜ ⎟ . . ⎜ ⎟ ⎛ ⎞ ⎜ ⎟ . . . ⎜ ⎟ ⎜ ⎟ ⎜ ⎟ = ⇒ = . . . ⎜ ⎟ ⎜ ⎟ 0 1 0 5 0 3 0 1 1 0 2 0 2 0 2 0 4 0 1 0 2 0 2 0 1 0 3 0 1 0 1 0 1 0 1 0 1 0 2 0 4 ⎜ ⎟ ⎜ ⎟ . . . . ⎜ ⎟ . . ⎜ ⎟ ⎜ ⎟ . . . . ⎝ ⎠ . . . . . . . ⎜ ⎟ ⎜ ⎟ ⎝ ⎠ 0 1 0 1 0 1 0 1 0 2 0 4 ⎜ ⎟ . . . . . . . ⎝ ⎠ 1 1 1 1 ⎛ ⎞ ⎜ ⎟ ⎜ ⎟ Thus,

1 1

0 7 0 1 1 50 0 25 ( ) S I Q

− −

. − . . . ⎛ ⎞ ⎛ ⎞ = − = = ⎜ ⎟ ⎜ ⎟

and

1 1 1 1 1 1 1 1 1 F ⎜ ⎟ ⎜ ⎟ ⎜ ⎟ ⎜ ⎟ = ⎜ ⎟ ( ) 0 2 0 6 0 50 1 75 S I Q = = = ⎜ ⎟ ⎜ ⎟ − . . . . ⎝ ⎠ ⎝ ⎠ 1 50 0 25 0 1 0 5 0.2 0 8 G S B . . . . . ⎛ ⎞⎛ ⎞ ⎛ ⎞ ⎜ ⎟⎜ ⎟ ⎜ ⎟

and

1 1 1 1 1 0 2 0 2 0 8 0 8 0 8 3 7 F = ⎜ ⎟ ⎜ ⎟ ⎜ ⎟ . . . . . ⎜ ⎟ ⎜ ⎟ 0 50 1 75 0 2 0 2 0 4 0.6 G S B = ⋅ = = ⎜ ⎟⎜ ⎟ ⎜ ⎟ . . . . . ⎝ ⎠⎝ ⎠ ⎝ ⎠ 1 3 0 4 0 4 0 6 0 6 0 6 3 7 ⎜ ⎟ ⎜ ⎟ . . . . . ⎜ ⎟ ⎝ ⎠

ΠΜΣ524: Μοντελοποίηση και Ανάλυση Απόδοσης Δικτύων (Ι. Σταυρακάκης / Α. Παναγάκης - ΕΚΠΑ) 58

SLIDE 30

Recurrent states and Limiting probabilities Recurrent states and Limiting probabilities

♣ Consider only an irreducible set of states. Theorem: Suppose X is irreducible and aperiodic. Then all states are recurrent non- null if and only if ( ) ( ) ( ) ( ) 1

i E j E

j i P i j j E j π π π

∈ ∈

= , , ∈ , =

∑ ∑

j

has a solution π . If there exists a solution π , then it is strictly positive, there are no

• ther solutions, and we have

( ) lim ( )

n n

j P i j i j E π

→∞

= , ,∀ , ∈ Corollary: If X in an irreducible aperiodic MC with finitely many states (no null Corollary: If X in an irreducible aperiodic MC with finitely many states (no-null states, no transient states), then 1 1 P π π π ⋅ = , ⋅ = has a unique solution. The solution π is strictly positive, and ( ) lim ( )

n n

j P i j π

→∞

= , , i j ∀ , .

59 ΠΜΣ524: Μοντελοποίηση και Ανάλυση Απόδοσης Δικτύων (Ι. Σταυρακάκης / Α. Παναγάκης - ΕΚΠΑ)

♣ A probability distribution π which satisfies P π π = ⋅ , is called an invariant distribution for X . ♣ If π is the initial distribution of X , that is, { } ( ), P X j j j E π = = ∈

then

{ }

n

P X j = = ( ) ( ) ( )

n i

i P i j j π π , =

∑

, for any n

E ∈ Proof:

2 n

P P P π π π π = ⋅ = ⋅ = = ⋅ L Algorithm: for finding lim ( )

n

P i j Algorithm: for finding lim ( )

n

P i j

→∞

,

• Consider the irreducible closed set containing j
• Solve for ( )

j π . Thus, we find lim ( )

n n

P j j

→∞

,

• For every i (not necessarily in E)

lim ( ) ( )lim ( )

n n n n

P i j F i j P j j

→∞ →∞

, = , , Compute ( ) F i j ,

• first. Then, find lim

( )

n n

P i j

→∞

,

ΠΜΣ524: Μοντελοποίηση και Ανάλυση Απόδοσης Δικτύων (Ι. Σταυρακάκης / Α. Παναγάκης - ΕΚΠΑ) 60

SLIDE 31

Example: E={1, 2, 3}, 0 3 0 5 0 2 0 6 0 4 0 4 P . . . ⎛ ⎞ ⎜ ⎟ = . . . ⎜ ⎟ ⎜ ⎟ ⎝ ⎠ 0 4 . . . ⎝ ⎠ (1) (1)0 3 (2)0 6 (2) (1)0 5 (3)0 4 P π π π = . + . (2) (1)0 5 (3)0 4 (3) (1)0 2 (2)0 4 (3)0 6 P π π π π π π π π π = ⇒ = . + + . = . + . + . 1 1 π = System’s Solution: ⎛ ⎞ 6 7 10 23 23 23 6 7 10 6 7 10 lim ( )

n

P P i j π

∞

⎛ ⎞ ⎜ ⎟ ⎜ ⎟ ⎛ ⎞ ⎜ ⎟ = ⇒ = = ⎜ ⎟ ⎜ ⎟ lim ( ) 23 23 23 23 23 23 6 7 10 23 23 23

n

P P i j π

→∞

⎜ ⎟ = ⇒ = , = ⎜ ⎟ ⎜ ⎟ ⎝ ⎠ ⎜ ⎟ ⎜ ⎟ ⎜ ⎟ ⎝ ⎠

61

23 23 23 ⎝ ⎠

ΠΜΣ524: Μοντελοποίηση και Ανάλυση Απόδοσης Δικτύων (Ι. Σταυρακάκης / Α. Παναγάκης - ΕΚΠΑ)

Example: ⎛ ⎞ 0 2 0 8 0 7 0 3 0 3 0 5 0 2 . . ⎛ ⎞ ⎜ ⎟ . . ⎜ ⎟ ⎜ ⎟ . . . ⎜ ⎟ E={1, 2, 3, 4, 5, 6, 7}, 0 6 0 4 0 4 0 6 0 1 0 1 0 2 0 2 0 3 0 1 P . . . ⎜ ⎟ = . . . ⎜ ⎟ ⎜ ⎟ . . . ⎜ ⎟ ⎜ ⎟ 0 1 0 1 0 2 0 2 0 3 0 1 0 1 0 1 0 1 0 1 0 2 0 4 . . . . . . . ⎜ ⎟ ⎜ ⎟ . . . . . . . ⎝ ⎠

1 1 2 2

0 3 0 5 0 2 0 2 0 8 7 8 6 7 10 0 6 0 4 0 7 0 3 15 15 23 23 23 0 4 0 6 P P π π . . . ⎛ ⎞ . . ⎛ ⎞ ⎛ ⎞ ⎛ ⎞ ⎜ ⎟ = ⇒ = , = . . . ⇒ = ⎜ ⎟ ⎜ ⎟ ⎜ ⎟ ⎜ ⎟ . . ⎝ ⎠ ⎝ ⎠ ⎝ ⎠ ⎜ ⎟ ⎝ ⎠ 0 4 0 6 . . . ⎝ ⎠ (6 1) (6 5) 0 2 0 2 0 8 0 8 0 8 ( ) ( ) F F , , . . . . . ⎡ ⎤ ⎡ ⎤ = ⎢ ⎥ ⎢ ⎥ ⎣ ⎦ ⎣ ⎦ L (7 1) (7 5) 0 4 0 4 0 6 0 6 0 6 F F ⎢ ⎥ ⎢ ⎥ , , . . . . . ⎣ ⎦ ⎣ ⎦ L

62 ΠΜΣ524: Μοντελοποίηση και Ανάλυση Απόδοσης Δικτύων (Ι. Σταυρακάκης / Α. Παναγάκης - ΕΚΠΑ)

SLIDE 32

Thus Thus, 7 8 15 15 7 8 ⎛ ⎞ ⎜ ⎟ ⎜ ⎟ ⎜ ⎟ 7 8 15 15 6 7 10 ⎜ ⎟ ⎜ ⎟ ⎜ ⎟ ⎜ ⎟ ⎜ ⎟ 23 23 23 6 7 10 lim 23 23 23

n n

P P

∞ →∞

⎜ ⎟ ⎜ ⎟ ⎜ ⎟ = = ⎜ ⎟ ⎜ ⎟ 6 7 10 23 23 23 1 4 1 6 4 8 5 6 8 ⎜ ⎟ ⎜ ⎟ ⎜ ⎟ ⎜ ⎟ . . . . ⎜ ⎟ 1 4 1 6 4 8 5 6 8 15 15 23 23 23 2 8 3 2 3 6 4 2 6 15 15 23 23 23 . . . . . . ⎜ ⎟ ⎜ ⎟ . . . . ⎜ ⎟ . . ⎜ ⎟ ⎝ ⎠ 15 15 23 23 23 ⎝ ⎠

ΠΜΣ524: Μοντελοποίηση και Ανάλυση Απόδοσης Δικτύων (Ι. Σταυρακάκης / Α. Παναγάκης - ΕΚΠΑ) 63

Example: ⎛ ⎞ Random walks: q p q p P q p ⎛ ⎞ ⎜ ⎟ ⎜ ⎟ = ⎜ ⎟ ⎜ ⎟ L L L ( X irreducible aperiodic (since state 0 is aperiodic)) q p ⎜ ⎟ ⎜ ⎟ ⎝ ⎠ M M M M O

1

p π ⎫ = ⎪

1 2 1 2 1 1 2 2 1 2

1 q q q p p p q p q p p q q

π π

π π π π π π π π

= =

⎪ ⎪ = + ⎫ ⎪ ⎛ ⎞ ⎪ = − / = = + ⎪ ⎛ ⎞ ⎜ ⎟ ⎪ = ⎝ ⎠ ⎬ ⎬ ⎜ ⎟

⇒ ⇒

L

2 2 1 3 2 2 3 3 2 3

1 q q p q q q p p p q q q q π π π π π = ⎝ ⎠ ⎬ ⎬ ⎜ ⎟ = + ⎝ ⎠ ⎪ ⎪ ⎛ ⎞ ⎪ ⎪ = − / = ⎭ ⎜ ⎟ ⎪ ⎝ ⎠ ⎪

⇒ ⇒

M M ⎪ ⎭ M M ♣ If p q ≥ : no solution of P π π = ⋅ , 1 1 π ⋅ = p q , ♣ If p q < :

( )( )

lim ( ) 1

j p p n n q q

P i j

→∞

, = −

64 ΠΜΣ524: Μοντελοποίηση και Ανάλυση Απόδοσης Δικτύων (Ι. Σταυρακάκης / Α. Παναγάκης - ΕΚΠΑ)

SLIDE 33

Example: Remaining lifetime ⎛ ⎞

1 2 3

1 1 p p p P ⎛ ⎞ ⎜ ⎟ ⎜ ⎟ = ⎜ ⎟ ⎜ ⎟ L L L ⎜ ⎟ ⎝ ⎠ M M M O

1 1

1 p π π π ν = + = ⎫

1 1 1 2 2 1 1 1 3 3 1 2 2 2

1 1 1 p p p p p p

π

π π π ν π π π ν π π π ν

=

+ ⎫ ⎪ = + + = − ⎪ ⎬ = + + = − − ⎪ ⎪

⇒

⎪ ⎭ M M M M Thus, ν

∞

=

∑

( ) ( ) ( ) p p p p p p + + + + + + + + +

j j

ν

=

=

∑

1 2 3 2 3 3

( ) ( ) ( ) p p p p p p + + + + + + + + + L L L L

=

1 2 3

2 3 p p p m + + + = L ♣ [ ]

n

m E Z = is the expected lifetime. ♣ If m = ∞ then all states are recurrent null and lim ( )

n

P i j =

65

♣ If m = ∞ then all states are recurrent null and lim ( )

n

P i j

→∞

, =

ΠΜΣ524: Μοντελοποίηση και Ανάλυση Απόδοσης Δικτύων (Ι. Σταυρακάκης / Α. Παναγάκης - ΕΚΠΑ)

Interpretation of Limiting Probabilities Proposition: Let j be an aperiodic recurrent non-null state, and let ( ) m j be the d i b h expected time between two returns to j . Then, 1 ( ) lim ( ) ( )

n n

j P j j m j π

→∞

= , = The limiting probability ( ) j π

• f being in state j is equal to the rate at which j is

visited. Proposition: Let j be an aperiodic recurrent non-null and let ( ) j π defined as p j p ( ) j

• previously. Then, for almost all ω ∈Ω

1 lim 1 ( ( )) ( ) 1

n j m n

X j n ω π

→∞

= + ∑ 1 m n

=

+ ♣. If f is a bounded function on E , then

n n

( ) ( ) 1 ( )

m j m m j E m

f X f j X

= ∈ =

=

∑ ∑ ∑

C ll X i d ibl t MC ith li iti b bilit Th f Corollary: X irreducible recurrent MC, with limiting probability π . Then, for any bounded function f on E : 1 lim ( ) ( ) ( )

n

f X f f j f j π π π = ⋅ , ⋅ =

∑ ∑

66

lim ( ) ( ) ( ) 1

m n m j E

f X f f j f j n π π π

→∞ = ∈

, + ∑

∑

ΠΜΣ524: Μοντελοποίηση και Ανάλυση Απόδοσης Δικτύων (Ι. Σταυρακάκης / Α. Παναγάκης - ΕΚΠΑ)

SLIDE 34

Similar results hold for expectations p Corollary: Suppose X is an irreducible recurrent MC with limiting distribution π . Then for any bounded function f on E 1 li [ ( )]

n

E f X f

∑

lim [ ( )] 1

i m n m

E f X f n π

→∞ =

= ⋅ + ∑ independent of i .

• If

( ) f j is the reward received whenever X is in j , then both the expected average If ( ) f j is the reward received whenever X is in j , then both the expected average reward in the long run and the actual average reward in the long run converge to the constant f π ⋅ . The ratio of the total reward received during the steps 0 1 n by using function f to The ratio of the total reward received during the steps 0 1… n , , , by using function f to the corresponding amount by using function g is ( ) lim

n m m

f X f π

=

⋅ =

∑

lim ( )

n n m m

g g X π

→∞ =

= ⋅

∑

♣ The same holds even in the case that X is only recurrent (can be null or periodic or ♣ The same holds even in the case that X is only recurrent (can be null or periodic or both)

ΠΜΣ524: Μοντελοποίηση και Ανάλυση Απόδοσης Δικτύων (Ι. Σταυρακάκης / Α. Παναγάκης - ΕΚΠΑ) 67

Th L t X b i d ibl t h i ith t iti t i P Th Theorem: Let X be an irreducible recurrent chain with transition matrix P . Then, the system P ν ν = ⋅ has a strictly positive solution; any other solution is a constant multiple of that one has a strictly positive solution; any other solution is a constant multiple of that one. Theorem: Suppose X is irreducible recurrent, and let ν be a solution of P ν ν = ⋅ . Then for any two functions f and g on E for which the two sums ( ) ( ) ( ) ( ) f i f i g i g i ν ν ν ν = =

∑ ∑

( ) ( ) ( ) ( )

i E i E

f i f i g i g i ν ν ν ν

∈ ∈

⋅ = , ⋅ =

∑ ∑

converge absolutely and at least one is not zero we have [ ( )]

n

E f X f ν

∑

[ ( )] lim [ ( )]

i m m n n i m m

E f X f g E g X ν ν

= →∞ =

⋅ = ⋅

∑ ∑

independently of i j E , ∈ . Moreover we also have ( ( )) lim ( ( ))

n m m n n m m

f X f g g X ω ν ν ω

= →∞ =

⋅ = ⋅

∑ ∑

for almost all ω ∈Ω

68 ΠΜΣ524: Μοντελοποίηση και Ανάλυση Απόδοσης Δικτύων (Ι. Σταυρακάκης / Α. Παναγάκης - ΕΚΠΑ)

SLIDE 35

Any non-negative solution of P ν ν = ⋅ is called an invariant measure of X . Comments: Any irreducible recurrent chain X has an invariant measure, and this is unique up y , q p to a multiplication by a constant. Furthermore, if X is also non-null, then 1 ( )

j

j ν ν ⋅ = ∑ is finite, and ν is a constant multiple of the limiting distribution π satisfying P π π = 1 1 π ⋅ = constant multiple of the limiting distribution π satisfying P π π = , 1 1 π ⋅ = The existence of an invariant measure ν for X does not imply that X is recurrent. For 1k f = 1 g = and i j = For 1k f , 1j g and i j

• 01 (

) ( ) lim ( ) 1 ( )

n j k m m n n

E X k j E X ν ν

= →∞

⎡ ⎤ ⎣ ⎦ = ⎡ ⎤

∑ ∑

( ) 1 ( )

j j m m

j E X ν

=

⎡ ⎤ ⎣ ⎦

∑

• ( )

( ) k j ν ν

is the ratio of the expected number of visits to k during the first n steps to the expected number of returns to j during the same period as n → ∞ the expected number of returns to j during the same period as n → ∞

• ( )

( ) k j ν ν

is the expected number of visits to k between two visits to state j

ΠΜΣ524: Μοντελοποίηση και Ανάλυση Απόδοσης Δικτύων (Ι. Σταυρακάκης / Α. Παναγάκης - ΕΚΠΑ) 69

Periodic States

It is sufficient to consider only an irreducible MC with periodic recurrent states. Lemma: Let X be an irreducible MC with recurrent periodic states with period δ . Then, the states can be divided into δ disjoint sets

1

B ,

2

B … , , Bδ such that ( ) P i j , = unless

1 2 2 3 1

• r
• r

. i B j B i B j B i B j B

δ

∈ , ∈ , ∈ , ∈ , ∈ , ∈ L

ΠΜΣ524: Μοντελοποίηση και Ανάλυση Απόδοσης Δικτύων (Ι. Σταυρακάκης / Α. Παναγάκης - ΕΚΠΑ) 70

SLIDE 36

Example: X MC with {1 2 3 4 5 6 7} E = , , , , , , 1 1 1 2 4 4 1 2 ⎛ ⎞ ⎜ ⎟ ⎜ ⎟ ⎜ ⎟ 1 2 3 3 1 2 3 3 ⎜ ⎟ ⎜ ⎟ ⎜ ⎟ ⎜ ⎟ ⎜ ⎟ 3 3 1 1 2 2 3 1 P ⎜ ⎟ ⎜ ⎟ ⎜ ⎟ = ⎜ ⎟ ⎜ ⎟ 3 1 4 4 1 1 ⎜ ⎟ ⎜ ⎟ ⎜ ⎟ ⎜ ⎟ ⎜ ⎟ 2 2 1 3 4 4 ⎜ ⎟ ⎜ ⎟ ⎜ ⎟ ⎜ ⎟ ⎝ ⎠ ⎝ ⎠ All states are periodic with period 3. The sets are

1

{1 2} B = , ,

2

{3 4 5} B = , , and

3

{6 7} B = , . >F B i t th MC h B i t t B d i th t B

ΠΜΣ524: Μοντελοποίηση και Ανάλυση Απόδοσης Δικτύων (Ι. Σταυρακάκης / Α. Παναγάκης - ΕΚΠΑ) 71

>From

1

B in one step the MC reaches

2

B , in two steps

3

B and in three steps

1

B . 23 25 71 121 48 48 192 192 11 7 29 43 ⎛ ⎞ ⎛ ⎞ ⎜ ⎟ ⎜ ⎟ ⎜ ⎟ ⎜ ⎟ ⎜ ⎟ ⎜ ⎟ 11 7 29 43 18 18 72 72 1 2 14 3 19 3 3 36 36 36 ⎜ ⎟ ⎜ ⎟ ⎜ ⎟ ⎜ ⎟ ⎜ ⎟ ⎜ ⎟ ⎜ ⎟ ⎜ ⎟ ⎜ ⎟ ⎜ ⎟

2 3

3 3 36 36 36 3 5 19 3 49 8 8 48 32 96 P P ⎜ ⎟ ⎜ ⎟ ⎜ ⎟ ⎜ ⎟ ⎜ ⎟ ⎜ ⎟ = , = ⎜ ⎟ ⎜ ⎟ ⎜ ⎟ ⎜ ⎟ 7 9 13 7 31 16 16 32 64 64 5 1 11 157 131 ⎜ ⎟ ⎜ ⎜ ⎟ ⎜ ⎜ ⎟ ⎜ ⎜ ⎟ ⎜ ⎜ ⎟ ⎜ ⎟ ⎟ ⎟ ⎟ ⎟ 12 8 24 288 288 3 1 9 111 81 8 16 16 192 192 ⎜ ⎟ ⎜ ⎜ ⎟ ⎜ ⎜ ⎟ ⎜ ⎜ ⎟ ⎜ ⎝ ⎠ ⎝ ⎠ ⎟ ⎟ ⎟ ⎟ 8 16 16 192 192 ⎝ ⎠ ⎝ ⎠ Note:

1 3 2

P P P P P P

⎛ ⎞ ⎜ ⎟ ⎜ ⎟ ⎜ ⎟ ⎜ ⎟ ⎜ ⎟ ⎜ ⎟

= , =

3

P

⎜ ⎟ ⎝ ⎠

• Chain corresponding to P has three closed sets

1

B ,

2

B ,

3

B and each one of these is irreducible, recurrent and aperiodic.

ΠΜΣ524: Μοντελοποίηση και Ανάλυση Απόδοσης Δικτύων (Ι. Σταυρακάκης / Α. Παναγάκης - ΕΚΠΑ) 72

p

• The previous limiting theory applies to compute

1

lim

m m P , 2

lim

m m P , 3

lim

m m P separately.

SLIDE 37

Th L t P th t iti t i f i d ibl MC ith t i di Theorem: Let P the transition matrix of an irreducible MC with recurrent periodic states of period δ , and let

1

B ,

2

B … , Bδ be as previously. Then, in the MC with transition matrix P Pδ = , the classes

1

B ,

2

B … , Bδ are irreducible closed sets of aperiodic states.

1 2

P P

δ ⎛ ⎞ ⎜ ⎟ ⎜ ⎟ ⎜ ⎟ ⎜ ⎟ 2

( ) ( )

a a

P P P i j P i j i j B P

δ δ ⎜ ⎟ ⎜ ⎟ ⎜ ⎟ ⎜ ⎟ ⎜ ⎟ ⎜ ⎟ ⎝ ⎠

= , , = , , , ∈ O Comments: If

a

i B ∈ , then { } 1 (mod )

i m b

P X B b a m δ ∈ = , = +

• (

)

n

P i j , does not have a limit as n → ∞ except when all the states are null (in which case ( )

n

P i j i j n , → , ∀ , , → ∞ ) The limits ( )

n m

P i j

δ +

exist as n → ∞ but are dependent on the initial The limits ( ) P i j , exist as n → ∞ , but are dependent on the initial state i .

ΠΜΣ524: Μοντελοποίηση και Ανάλυση Απόδοσης Δικτύων (Ι. Σταυρακάκης / Α. Παναγάκης - ΕΚΠΑ) 73

Th L P d B i l d h h h i i ll Th Theorem: Let P and

a

B as previously and suppose that the chain is non-null. Then, for any {0 1 1} m … δ ∈ , , , − ( ) (mod ) lim ( )

a b n m

j i B j B b a m P i j

δ

π δ

+

∈ , ∈ , = + ⎧ ⎨ lim ( )

• therwise

n m n

P i j

δ →∞

, = ⎨ ⎩ The probabilities ( ) j π , j E ∈ form the unique solution of ( ) ( ) ( ) ( ) j i P i j i π π π δ

∑ ∑

( ) ( ) ( ) ( )

i E i E

j i P i j i π π π δ

∈ ∈

= , , =

∑ ∑

ΠΜΣ524: Μοντελοποίηση και Ανάλυση Απόδοσης Δικτύων (Ι. Σταυρακάκης / Α. Παναγάκης - ΕΚΠΑ) 74

SLIDE 38

0 5 0 5 0 4 0 6 . . ⎛ ⎞ ⎜ ⎟ . . ⎜ ⎟ Example: Let X be a MC with state space {1 2 3 4 5} E = , , , , , 1 0 8 0 2 1 P ⎜ ⎟ ⎜ ⎟ = . . ⎜ ⎟ . . . ⎜ ⎟ ⎜ ⎟ ⎝ ⎠ 1 ⎜ ⎟ . . . ⎝ ⎠ The chain is irreducible, recurrent non-null periodic with period 2 δ = . 0 4 0 5 0 1 . . . ⎛ ⎞ ⎜ ⎟

2

0 32 0 6 0 08 1 0 4 0 6 P P ⎛ ⎞ ⎜ ⎟ . . . ⎜ ⎟ ⎜ ⎟ = = . . . ⎜ ⎟ ⎜ ⎟

( ) ( )

1 2

0 32 0 60 0 08 0 4 0 6 π π = . . . , = . . 0 4 0 6 0 4 0 6 . . ⎜ ⎟ ⎜ ⎟ . . ⎝ ⎠ 0 32 0 60 0 08 ⎛ ⎞ 0 4 0 6 ⎛ ⎞

2

0 32 0 60 0 08 0 32 0 60 0 08 lim 0 32 0 60 0 08

n

P

→

. . . ⎛ ⎞ ⎜ ⎟ . . . ⎜ ⎟ ⎜ ⎟ = . . . ⎜ ⎟

2 1

0 4 0 6 0 4 0 6 lim 0 4 0 6

n

P

+ →

. . ⎛ ⎞ ⎜ ⎟ . . ⎜ ⎟ ⎜ ⎟ = . . ⎜ ⎟ 0 4 0 6 0 4 0 6

n→∞

⎜ ⎟ . . ⎜ ⎟ ⎜ ⎟ . . ⎝ ⎠ 0 32 0 60 0 08 0 32 0 60 0 08

n→∞

⎜ ⎟ . . . ⎜ ⎟ ⎜ ⎟ . . . ⎝ ⎠

ΠΜΣ524: Μοντελοποίηση και Ανάλυση Απόδοσης Δικτύων (Ι. Σταυρακάκης / Α. Παναγάκης - ΕΚΠΑ) 75

1 ⎛ ⎞ Example: Random Walks ( p q < ) 1 q p P q p ⎛ ⎞ ⎜ ⎟ ⎜ ⎟ = ⎜ ⎟ ⎜ ⎟ ⎝ ⎠ O O ⎜ ⎟ ⎝ ⎠ O O

• Cyclic Classes

1

{0 2 4 } B … = , , , ,

2

{1 3 5 } B … = , , ,

• Invariant solution

P ν ν = ⋅

• Invariant solution

P ν ν = ⋅

2 4 2 2 2 4

1 1 p p … q q p p ν ν ν = , = , = ,

1 3 5 3 5

1 p p … q q q ν ν ν = , = , = , Normali e:

2

1 1 1 2 1 1 1 p p ⎡ ⎤ + + + + + ⎢ ⎥

∑

Normalize:

2

1 1 1 1 1

i p p q q

p p q q q q ν = + + + + = + = ⎢ ⎥ − − ⎣ ⎦

∑

L M lti l h t b 1

p (

2

∑

)

( )

2 4

… π π π , , , = (

)(

)

3 2 4

1 1

p p p q

… − , , , Multiply each term by 1

p q

− . ( 2

i

ν =

∑

)

( )

2 4

, , ,

( )(

)

2 4

q q q

, , ,

( )

1 3 5

… π π π , , , = (

)(

)

2 4 3 5

1

1

p p p q q q q

… − , , ,

76 ΠΜΣ524: Μοντελοποίηση και Ανάλυση Απόδοσης Δικτύων (Ι. Σταυρακάκης / Α. Παναγάκης - ΕΚΠΑ)

SLIDE 39

Hence,

3 2 4 2

1 1 p p q q p ⎛ ⎞ ⎜ ⎟ ⎜ ⎟ ⎜ ⎟ L

2 3 3

1 p q q p p ⎛ ⎞ ⎜ ⎟ ⎜ ⎟ ⎜ ⎟ L

3 2 3 2 4

1 lim 1 1

n n

p q q p P p p q

→∞

⎜ ⎟ ⎜ ⎟ ⎜ ⎟ ⎛ ⎞⎜ ⎟ = − ⎜ ⎟⎜ ⎟ ⎝ ⎠⎜ ⎟ L L

2 4 2 1 2 3

1 lim 1 1

n n

p p q q p P p q

+ →∞

⎜ ⎟ ⎜ ⎟ ⎜ ⎟ ⎛ ⎞⎜ ⎟ = − ⎜ ⎟⎜ ⎟ ⎝ ⎠⎜ ⎟ L L

2 4 2 3

1 q q q p q q ⎝ ⎠⎜ ⎟ ⎜ ⎟ ⎜ ⎟ ⎜ ⎟ ⎜ ⎟ L

3 3 2 4

1 q q q p p q q ⎝ ⎠⎜ ⎟ ⎜ ⎟ ⎜ ⎟ ⎜ ⎟ ⎜ ⎟ L q q ⎜ ⎟ ⎝ ⎠ M M M M M O q q ⎜ ⎟ ⎝ ⎠ M M M M M O

ΠΜΣ524: Μοντελοποίηση και Ανάλυση Απόδοσης Δικτύων (Ι. Σταυρακάκης / Α. Παναγάκης - ΕΚΠΑ) 77

Transient States If a MC has only finitely many transient states, then it will eventually leave the set of transient states never to return. If there are infinitely many transient states, it is possible for the chain to remain in the set of transient states forever. Example: ⎛ ⎞

1 2 3 1 2 1

p p p p p p p P p p ⎛ ⎞ ⎜ ⎟ ⎜ ⎟ ⎜ ⎟ = ⎜ ⎟ L L L

1

p ⎜ ⎟ ⎜ ⎟ ⎜ ⎟ ⎝ ⎠ L M M M M O All states are transient If initial state is i , then the chain stays forever in the set { 1 2 } i i i … , + , + , . As n → ∞ ( ) X ω → ∞ As n → ∞ , ( )

n

X ω → ∞

ΠΜΣ524: Μοντελοποίηση και Ανάλυση Απόδοσης Δικτύων (Ι. Σταυρακάκης / Α. Παναγάκης - ΕΚΠΑ) 78

SLIDE 40

Let A E ⊂ , Q the matrix obtained from P by deleting all the rows and columns corresponding to states which are not in A . Then, for i j A , ∈ ( )

n

Q i j , =

1 1

1 1 2 1

( ) ( ) ( )

n

n i A i A Q i i Q i i

Q i j

−

− ∈ ∈

, , ,

∑ ∑

L L =

{ }

1 1 i n n

P X A X A X j

−

∈ , , ∈ , = L ( )

n

Q i j

∑

{ }

P X A X A X A ( )

n j A

Q i j

∈

, =

∑

{ }

1 1 i n n

P X A X A X A

−

∈ , , ∈ , ∈ L The event

1 1

{ }

n

X A … X A

+

∈ , , ∈ is a subset of

1

{ }

n

X A … X A ∈ , , ∈ , therefore

1

( ) ( )

n n j A j A

Q i j Q i j

+ ∈ ∈

, ≥ ,

∑ ∑

j j

Let ( ) lim ( )

n

f i Q i j i A = , , ∈

∑

( ) ( )

n j A

f Q j

→∞ ∈

, ,

∑

♣ ( ) f i is the probability that starting at i A ∈ , the chain stays in the set A forever.

ΠΜΣ524: Μοντελοποίηση και Ανάλυση Απόδοσης Δικτύων (Ι. Σταυρακάκης / Α. Παναγάκης - ΕΚΠΑ) 79

Proposition: The function f is the maximal solution of the system 1 h Qh h = , ≤ ≤ Q , Either f =

• r sup

( ) 1

i A f i ∈

= ♣ An application of the previous proposition was given in a theorem on the classification of states: classification of states: Theorem: Let X an irreducible MC with transition matrix P , and let Q be the matrix obtained from P by deleting the k -row and k -column for some k E ∈ . Then all states are recurrent if and only if the only solution of all states are recurrent if and only if the only solution of ( ) ( ) ( ) ( ) 1

j E

h i Q i j h j h i i E

∈

= , , ≤ ≤ , ∈

∑

is ( ) h i = for all i E ∈ . { } E E k = − . is ( ) h i for all i E ∈ . { } E E k . Proof: Fix a perticular state and name it 0 . Since X is irreducible it is possible to go from 0 to some {0} i A E ∈ = − . Since X is irreducible it is possible to go from 0 to some {0} i A E ∈ . If the probability ( ) f i of remaining in A forever is ( ) f i = for all i A ∈ , then with probability 1, the chain will leave A and enter 0 again. Hence if the only solution of the system is h = then state 0 is recurrent Hence, if the only solution of the system is h = , then state 0 is recurrent , and that in turn implies that all states are recurrent. Conversely, if all states are recurrent, then the probability of remaining in the set A forever must be zero, since 0 will be reached with probability one

ΠΜΣ524: Μοντελοποίηση και Ανάλυση Απόδοσης Δικτύων (Ι. Σταυρακάκης / Α. Παναγάκης - ΕΚΠΑ) 80

, p y from any state i A ∈

SLIDE 41

Example: (Random Walk) p ⎛ ⎞ ⎜ ⎟ p q p Q q p ⎜ ⎟ ⎜ ⎟ = ⎜ ⎟ ⎜ ⎟ ⎝ ⎠ O ⎝ ⎠ O

• If p

q > all states are transient.

i

( ) 1 1 2 3

i

q f i i … p ⎛ ⎞ = − , = , , , ⎜ ⎟ ⎝ ⎠ This is the maximal solution since sup ( ) 1 f i = This is the maximal solution since sup ( ) 1

i f i

. Interpretation: Starting at a state k (e.g. 7 k = ) the probability of staying forever within the set {1 2 3 } i l t

( )

7

1

q

{1 2 3 } … , , , is equal to

( )

1

q p

− . If k k

′ >

, the probability of remaining in {1 2 3 } … , , , is greater. From the shape of P : the restriction of P to the set { 1 } k k … , + , is the same as the p { } , , matrix Q . Hence, for all {1 2 3 } k … ∈ , , ,

{ }

1 1 2

1

i k i

q P X k X k …

+ +

⎛ ⎞ ≥ , ≥ , = − ⎜ ⎟ ⎝ ⎠

ΠΜΣ524: Μοντελοποίηση και Ανάλυση Απόδοσης Δικτύων (Ι. Σταυρακάκης / Α. Παναγάκης - ΕΚΠΑ) 81

{ }

1 2 k i

p

+

⎜ ⎟ ⎝ ⎠ For any subset A of E , let ( )

A

f i the probability of remaining forever in A given the y , ( )

A

f p y g g initial state i A ∈ . Then, If A is an irreducible recurrent class, 1

A

f = . If A is a proper subset of an irreducible recurrent class f = If A is a proper subset of an irreducible recurrent class,

A

f = . If A is a finite set of transient states,

A

f = . If A is an infinite set of transient states, then either

A

f =

• r

A

f ≠ . In the latter case the chain travels through a sequence of sets (

1 2 3

A A A ⊃ ⊃ L) to “infinite”.

ΠΜΣ524: Μοντελοποίηση και Ανάλυση Απόδοσης Δικτύων (Ι. Σταυρακάκης / Α. Παναγάκης - ΕΚΠΑ) 82