02407 Stochastic Processes Outline Discrete time Markov chains Outline Lecture 4: Outline The period of a state The period of a state Random walks Random walks ■ The period of a state Classification of states Classification of states Discrete-time Markov Chains II Stationary Stationary ■ Recap classification of states: Transient, Null Recurrent, Positive distributions distributions A criterion for A criterion for Recurrent transience transience Limiting distributions Limiting distributions ■ Partition the state space S into transient states, and closed Summary Summary irreducible sets of persistent states. Next week Next week Exercises Exercises ■ Stationary distributions ... Existence ... relation to mean recurrence times Uffe Høgsbro Thygesen Informatics and Mathematical Modelling Technical University of Denmark 2800 Kgs. Lyngby – Denmark Email: uht@imm.dtu.dk 1 / 23 2 / 23 The period of a state Transience vs. persistency/recurrency Outline Let the chain start in state i . Outline As before, let N i be the number of visits to state i , i.e. The period of a state The period of a state N i = # { n ≥ 1 : X n = i } . Likewise, let T i be the time of first Random walks Random walks Consider the times where a revisit is possible Classification of states Classification of states (re)visit to state i , i.e. T i = min { n ≥ 1 : X n = i } . Stationary Stationary distributions distributions State Definition Criterion { n : p ii ( n ) > 0 } A criterion for A criterion for transience transience P i ( N i > 0) < 1 Transient � n ≥ 1 p ii ( n ) < ∞ Limiting distributions Limiting distributions The greatest common divisor of these times are the period of state i . P i ( N i > 0) = 1 � Summary Summary n ≥ 1 p ii ( n ) = ∞ Null persistent Next week Next week E i T i = ∞ lim sup n →∞ p ii ( n ) = 0 Exercises Examples: In a simple (a)symmetric random walk, all states are Exercises P i ( N i > 0) = 1 � n ≥ 1 p ii ( n ) = ∞ periodic with period 2. Positive recurrent E i T i < ∞ lim sup n →∞ p ii ( n ) > 0 In the growth/rest process, all states are aperiodic . For a transient state i : N i is geometrically distributed and T i is infinite with non-zero probability (under P i ). For a persistent state, T i is finite w.p. 1, and N i is infinite w.p. 1 (under P i ). 3 / 23 4 / 23
State classification in general random walk Random walks in higher dimensions Outline Outline A vector process in N dimensions X n = ( X (1) n , . . . , X ( N ) ) , where the S n +1 = S n + X n , S 0 = 0 n The period of a state The period of a state co-ordinate processes X ( i ) Random walks Random walks are independent biased random walks with n where X n are i.i.d. with mean µ and variance σ 2 . Classification of states Classification of states mean µ i and variance σ 2 Stationary Stationary i . distributions distributions According to the Central Limit Theorem A criterion for A criterion for transience transience � � Limiting distributions Limiting distributions µ 2 1 − n µ 2 1 � � − n Summary Summary � i → p 00 ( n ) ≈ (2 πn ) N/ 2 � σ i exp ∼ N ( nµ, nσ 2 ) S n so p 00 ( n ) ≈ √ 2 πnσ 2 exp σ 2 Next week Next week 2 σ 2 2 i Exercises Exercises i The origin is null persistent iff ∀ i : µ i = 0 and n ≤ 2 ; otherwise Corollary (4) (p. 221) says that the state 0 is persistent iff transient. � n p 00 ( n ) = ∞ . I.e., iff µ = 0 . (The conclusion can be generalized to the situation where the Compare exercise 6.3.2 where we had X n ∈ {− 1 , 2 } . co-ordinate processes are correlated) 5 / 23 6 / 23 Connecting random walks and diffusion Survival analysis with competing hazards Outline Compare the diffusion equation in C ( x, t ) : Outline A model of the course of university studies of a single student: The period of a state The period of a state Random walks Random walks ∂t = D∂ 2 C X n ∈ N ∪ { G } ∪ { A } ∂C Classification of states Classification of states Stationary Stationary ∂x 2 distributions distributions A criterion for A criterion for transience transience with a symmetric random walk S n with p i ( i +1) = p i ( i − 1) = p , Limiting distributions Limiting distributions where X n = i means that the student is at his/her i th semester, Summary Summary p ii = 1 − 2 p . X n = G means graduated, Next week Next week Exercises Exercises When n is large, S n is approximately distributed as N (0 , σ 2 n ) . X n = A means studies abandoned without graduation. 1. Compare this with the solution of the diffusion equation with Transition probabilities: initial condition C ( x, 0) = δ ( x ) p iG = g i p iA = a i p i ( i +1) = 1 − g i − a i x 2 1 exp( − 1 C ( x, t ) = √ 2 Dt ) 2 2 π 2 Dt We can divide the state space into three: The transient states N , the absorbing state G and the absorbing state A . 2. Compare the forward equation for the random walk with the second-order central finite difference scheme for discretisation of the diffusion equation. (If you have studied numerics ...) 7 / 23 8 / 23
Classification of state space Classification of state space Outline Outline The period of a state The period of a state C1 Random walks Random walks Classification of states Classification of states Stationary Stationary distributions distributions A criterion for A criterion for transience transience Limiting distributions Limiting distributions Summary Summary Next week Next week Exercises Exercises C3 C2 T 9 / 23 10 / 23 Communicating states Closed and irreducible subsets of state space Outline State i communicates with j (written i → j ), if p ij ( m ) > 0 for some Outline A closed set C is one which can never be departed, once entered: The period of a state The period of a state m . p ij = 0 for all i ∈ C and all j �∈ C . Random walks Random walks Classification of states Classification of states Stationary In terms of the graph: If there is a path from i to j . Stationary An irreducible set C is one in which all states intercommunicate. distributions distributions A criterion for A criterion for i and j intercommunicates (written i ↔ j ) if i → j and j → i . (If The Decomposition theorem says that state space can be transience transience Limiting distributions Limiting distributions there is a closed path containing both i and j ). partitioned into the transient states T and closed irreducible sets C i : Summary Summary Next week Next week ↔ is an equivalence relation on state space S . Exercises Exercises S = T ∪ C 1 ∪ C 2 ∪ · · · Two intercommunicating states must have the same qualitative When studying long-time behaviour, we can concentrate on the cases properties (theorem 2): Same period, same persistency. S = T and S = C 1 . 11 / 23 12 / 23
Stationary distributions The symmetric simple random walk We had The stationarity equation π = π P reads Outline Outline The period of a state The period of a state µ n +1 = µ n P Random walks Random walks π j = 1 Classification of states Classification of states 2( π j − 1 + π j +1 ) Stationary A distribution π on S (such that π j ≥ 0 and � j π j = 1 ) is Stationary distributions distributions A criterion for stationary iff A criterion for transience transience on the interior. The general solution is π = π P Limiting distributions Limiting distributions Summary Summary Next week Next week π j = aj + b Exercises Exercises 1. With two reflecting barriers, π j = b on the interior. 2. With no barriers, no distribution can live up to this. So what can we do with the solution π j = 1 ? This is important: In many applications, we only care about stationary distributions. 13 / 23 14 / 23 The asymmetric simple random walk A queue application Outline The stationarity equation π = π P reads Outline A single server handles customers one at a time. The period of a state The period of a state Random walks Random walks There are two types of events : New customers arriving, and the π j = π j − 1 p + π j +1 q Classification of states Classification of states Stationary Stationary completion of service of a customer. distributions distributions A criterion for A criterion for on the interior. The general solution is S n models the queue length immediately after event no. n . transience transience Limiting distributions Limiting distributions Summary � j Summary When the queue is non-empty, the next event is either a new � p Next week Next week π j = c · + k customer arriving (with probability p ) or a customer departing (w.p. Exercises Exercises q q = 1 − p ). With p < 1 Result: A stationary queue length distribution exists iff the mean 2 and a lower reflecting barrier at 0, we find k = 0 , and a service time is smaller than the mean interarrival time. geometrically decaying stationary distribution π . If the two times are equal, then S n is null recurrent. If the mean service time is the greater, then all states are transient. The queue length grows to infinity. 15 / 23 16 / 23
Recommend
More recommend
