Markov Chains: Classification of States IE 502: Probabilistic - - PowerPoint PPT Presentation

IE502: Probabilistic Models IEOR @ IITBombay

Markov Chains:

Classification of States

IE 502: Probabilistic Models Jayendran Venkateswaran IE & OR

IE502: Probabilistic Models IEOR @ IITBombay

Classification of States

• State j is accessible from state i iff, starting in i, it

is possible to enter j at some time (or step).

• Two states i and j that are accessible to each
• ther are said to communicate (i ↔ j)
• Communication is an equivalence relation

– i ↔ i – If i ↔ j then j ↔ i – If i ↔ j and j ↔ k then i ↔ k

IE502: Probabilistic Models IEOR @ IITBombay

Classification of States: Class

• Two states that communicate are in same class.

– MC can have 1 or more classes. – Classes are either disjoint or identical.

• If the Markov chain has only one class, then the

chain is irreducible

– That is, all states communicate with one another

• Identify the classes of the following MCs:

0.4 0.6 0.2 0.7 0.1 0.5 0.5 0.4 0.6 0.7 0.3 0.2 0.4 0.4

IE502: Probabilistic Models IEOR @ IITBombay

Classification of States: Absorbing State

• State i is called absorbing if it is impossible to

leave it (i.e. pii = 1).

• A Markov chain is absorbing if it has at least one

absorbing state, and if from every state it is possible to go to an absorbing state (not necessarily in one step)

– When a process reaches an absorbing state, then it said to be absorbed.

IE502: Probabilistic Models IEOR @ IITBombay

Classification of States: Transient and Recurrent states

• State i is said to be transient if the probability to

return to i is less than 1.

– That is, State i is transient if, starting in state i, there is a non-zero probability that we never return to i

• State i is said to be recurrent if the probability

that, starting in state i, the process will ever return to state i is 1.

IE502: Probabilistic Models IEOR @ IITBombay

Transient and Recurrent states (2)

• If state i is transient then, starting in i, the process

visits state i finite number of times

• If state i is recurrent then, starting in i, the process

will reenter state i again and again, infinitely often.

• Recurrence is a class property

 Transience is also a class property!

• In a finite state MC, all states cannot be transient!

IE502: Probabilistic Models IEOR @ IITBombay

Classification of States: Periodicity

• State i has period d if, starting in state i, any return

to i must occur in multiples of d time steps

• State with period 1 is said to be aperiodic.
• Determine period of state 0 in the following chains
• Periodicity is a class property

– If state i has period d and i ↔ j then state j has period d

• Markov chain is aperiodic if all states are aperiodic

1 2 1 1 p 1-p 1 2 3 1 1/2 1/2 1/2 1/2 1/2 1/2

IE502: Probabilistic Models IEOR @ IITBombay

Classification of States: Ergodicity

• If state i is recurrent, and (starting in i) the mean

time to return to i is finite, then it is said to be positive recurrent.

• If mean time to return is infinite, then it is null
• recurrent. (Example null recurrent chain)
• Positive recurrence is a class property

– In a finite state Markov chain all recurrent states are positive recurrent

1 2 3 1 1/2 2/3 1/2 1/3 1/4 . . .

IE502: Probabilistic Models IEOR @ IITBombay

Classification of States: Ergodicity (2)

• Positive recurrent, aperiodic states are called as

ergodic

• Ergodicity is a class property
• If all states are ergodic then the chain is called as

Ergodic Markov Chain

IE502: Probabilistic Models IEOR @ IITBombay

Example

• Consider the chain:
• Identify classes: are they transient or recurrent?
• Comment on the periodicity of the classes

– Is the Markov Chain periodic or aperiodic?

• Comment on the ergodicity of the classes

– Is the Markov Chain ergodic? 1/2 1/2 3/4 1/4 1 1 1/4 1/4 1/2

IE502: Probabilistic Models IEOR @ IITBombay

Terminology

• " Transition into state i "

– enter state i from some other state – Change or move from other state to state i

• " Transition out of state i "

– Leave state i and go to some other state – Change or move to another state from state i

• Stages, Steps, Round of a game, time step and

time could be used interchangeably. Learn to interpret correctly given the problem context.