Outlines Stochastic Process Discrete Time Markov Chain (DTMC) 2 - - PowerPoint PPT Presentation

Markov Chains (1)

2

Outlines

 Stochastic Process  Discrete Time Markov Chain (DTMC)

3

Stochastic Process

 A stochastic process is a family of random variables

, defined on a given probability space, indexed by parameter t, where t varies over an index set T.

 Thus the above family of random variables is a family of functions

.

 For a fixed

is a random variable (denoted by ) as s varies over the sample space S.

 For a fixed sample point

the expression is a single function of time t, called a sample function or a realization

• f the process.

{ ( )| } X t t T  { ( , )| , } X t s s S t T  

1

1 1

, ( ) ( , )

t

t t X s X t s  

1

( ) X t

1

s S 

1

1

( ) ( , )

s

X t X t s 

4

Stochastic Process…

 If the sample space of a stochastic process is discrete,

then it is called a discrete-state process, often referred to as a chain.

 Alternatively if the state space is continuous, then we

have a continuous-state process.

 Similarly, if the index set T is discrete, then we have a

discrete-time process; otherwise we have a continuous- time process.

 A discrete-time process is also called a stochastic

sequence and is denoted by .

{ | }

n

X n T 

5

Stochastic Process…

 For a fixed time

, the term is a simple random variable that describe the state of the process at time .

 For a fixed number

, the probability of the event gives the CDF of the random variable , denoted by



is known as the first-order distribution of the process .

 Given two time instants

and , and are two random variables on the same probability space. Their joint distribution is known as the second-order distribution of the process and is given by

1

( ) X t

1

t t 

1

t

1

x

1 1

( ) X t x 

1

( ) X t

1

1 1 ( ) 1 1 1

( ; ) ( ) [ ( ) ]

X t

F x t F x P X t x   

1 1

( ; ) F x t { ( )| 0} X t t 

1

t

2

t

1

( ) X t

2

( ) X t

1 2 1 2 1 1 2 2

( , ; , ) [ ( ) , ( ) ] F x x t t P X t x X t x   

6

Stochastic Process…

 In general, we define the nth-order joint distribution of

the stochastic process by

 A stochastic process

is said to be stationary in the strict sense if for , its nth-order joint CDF satisfies the condition for all vectors and , and all scalars such that

1 1

( ; ) [ ( ) ,..., ( ) ]

n n

F x t P X t x X t x    ( ), X t t T  { ( )| } X t t T  1 n  ( ; ) ( ; ) F x t F x t   

n

x

n

t T  

n i

t T   

7

Stochastic Process…

 A stochastic process

is said to be an independent process provided its nth-order joint distribution satisfies the condition

 A

renewal process is defined as a discrete-time independent process where are independent identically distributed (i.i.d), nonnegative random variables.

1 1

( ; ) ( ; ) [ ( ) ]

n n i i i i i i

F x t F x t P X t x

 

  

 

{ ( )| } X t t T  { | 1,2,...}

n

X n 

1 2

, ,... X X

8

Stochastic Process…

 Though the assumption of an independent process

considerably simplifies analysis, such an assumption is

• ften unwarranted, and we are forced to consider some

sort of dependence among these random variables.

 The simplest and the most important type of dependence

is the first-order dependence or Markov dependence.

 A stochastic process

is called a Markov process if for any , the conditional distribution of for given values of depends only on :

1 1

[ ( ) | ( ) , ( ) ,... ( ) ]

n n n n

P X t x X t x X t x X t x

 

   

{ ( )| } X t t T 

1 2

...

n

t t t t t     

1

( ), ( ),..., ( )

n

X t X t X t ( ) X t ( )

n

X t

[ ( ) | ( ) ]

n n

P X t x X t x   

9

Stochastic Process…

 In many problems of interest, the conditional distribution

function mentioned in definition of Markov process has the property of invariance with respect to the time origin

 In this case, the Markov chain is said to be (time-)

homogeneous.

 For a homogeneous Markov chain, the past history of

the process is completely summarized in the current state; therefore, the distribution for the time Y the process spends in a given state must be memory less.

n

t

[ ( ) | ( ) ] [ ( ) | (0) ]

n n n n

P X t x X t x P X t t x X x      

10

Discrete Time Markov Chain (DTMC)

 We choose to observe the state of a system at a discrete

set of time points.

 The

successive

• bservations

define the random variables at time steps 0, 1, 2, …, n

• respectively. If

, then the state of the system at time step n is j. is the initial state of the system. The Markov property can then be succinctly stated as

 The above equation implies that given the present state

• f the system, the future is independent of its past.

1 1 1 1 1 1

( | , ,..., ) ( | )

n n n n n n n n

P X i X i X i X i P X i X i

   

      

1 2

, , ,...,

n

X X X X

n

X j  X

11

Discrete Time Markov Chain (DTMC)…



denotes the pmf of the random variable

 We will only be concerned with homogenous Markov

• chains. For such chains, we use the following notation to

denote n-step transition probabilities.

 The one-step transition probabilities

are simply written as , thus:

( ) ( )

j n

p n P X j   ( )

j

p n ( ) ( | )

jk m n m

p n P X k X j



  

1

(1) ( | )

jk jk n n

p p P X k X j



    (1)

jk

p

jk

p

12

Discrete Time Markov Chain (DTMC)…

 The pmf of the random variable

, often called the initial probability vector, is specified as

 The one-step transition probabilities are compactly specified

in the form of a transition probability matrix

 The entries of the matrix P satisfy the following two properties

1

p(0) [ (0), (0),...] p p 

X

00 01 02 10 11 12

. . P [ ] . . . . . . . .

ij

p p p p p p p               , ; i j I  and . i I  1,

ij

p   1,

ij j I

p





