Outline Outline Markov Processes Markov Processes Important - - PowerPoint PPT Presentation

SLIDE 1

1

• G. Ahmadi

ME 529 - Stochastics

• G. Ahmadi

ME 529 - Stochastics

Outline Outline

• Markov Processes

Markov Processes

• Important Properties

Important Properties

• Chapman

Chapman-

• Kolmogorov

Kolmogorov-

• Equation

Equation

• Fokker

Fokker-

• Planck Equation

Planck Equation

• Fokker

Fokker-

• Planck Equation for Vector

Planck Equation for Vector Markov Processes Markov Processes

• Fokker

Fokker-

• Planck Equation for Ito’s

Planck Equation for Ito’s Equation Equation

• G. Ahmadi

ME 529 - Stochastics

A stochastic process X(t) is called a Markov A stochastic process X(t) is called a Markov process if for every n and any t process if for every n and any t1

1 < t

< t2

2 < … <

< … < t tn

n ,

, its conditional probability satisfies the its conditional probability satisfies the following condition following condition

.

• r
• r

( ) ( ) ( ) ( ) { } ( ) ( ) { }

1 1 2 1

| ,..., , |

− − −

≤ = ≤

n n n n n n n

t X x t X P t X t X t X x t X P

( ) ( ) { } ( ) ( ) { }

1 1

| |

− −

≤ = ≤ ≤

n n n n n n

t X x t X P t t all for t X x t X P

• G. Ahmadi

ME 529 - Stochastics

i) i) X(t) is also Markov in reverse: X(t) is also Markov in reverse: ii) ii) The future is independent of the past under The future is independent of the past under the given condition of the present. the given condition of the present. iii) iii) If for any t If for any t1

1 < t

< t2

2, X(t

, X(t2

2)

) – – X(t X(t1

1) is independent of

) is independent of X(t) for t X(t) for t ≤ ≤ t t1

1, the

, the X(t X(t) is a Markov process. ) is a Markov process. Thus, independent increment processes Thus, independent increment processes (Poisson, Wiener (Poisson, Wiener-

• Levy) are Markov processes.

Levy) are Markov processes.

.

( ) ( ) { } ( ) ( ) { }

2 1 1 2 1 1

| | t X x t X P t t all for t X x t X P ≤ = ≥ ≤

SLIDE 2

2

• G. Ahmadi

ME 529 - Stochastics

.

iv) iv) v) v) X(t X(t) is associated with a 1 ) is associated with a 1st

st order equation,

• rder equation,

with solution with solution vi) vi) Conditional probability density satisfies Conditional probability density satisfies

( ) ( )

t j t X dt dX = − , β

( )

dt dW t j =

( ) ( ) ( ) ( ) ( )

∫ ∫

+ + =

t t t t

d j d X t X t X , τ τ τ τ τ β

( ) ( )

1 1 1 1 2 2 1 1

, | , , ; , ;...; , | ,

− − − −

=

n n n n X n n n n

t x t x f t x t x t x t x f X

( ) ( ) ( ) { } ( ) ( ) { }

1 1 1

| ,..., |

− −

=

n n n n

t X t X E t X t X t X E

• G. Ahmadi

ME 529 - Stochastics

.

vi) vi) The following relation holds The following relation holds If then If then A Markov Process is fully specified if: A Markov Process is fully specified if: a) Given 1 a) Given 1st

st order density and transition

• rder density and transition

probability density; probability density; b) 2 b) 2nd

nd order density;

• rder density;

c) Transition density and X(0) c) Transition density and X(0)

( ) ( ) ( ) ( )

1 1 1 1 2 2 2 2 3 3 3 3 2 2 1 1

, , | , , | , , ; , ; , t x f t x t x f t x t x f t x t x t x f

X X X

=

X

( ) ( ) ( ) ( )

1 1 1 1 2 2 1 1 1 1

, , | , ... , | , , ;...; , t x f t x t x f t x t x f t x t x f

X X n n n n X n n − −

=

X

( )

• x

X =

( ) ( )

1 1 1 1

, x | t , x f t , x f

• X

X

=

• G. Ahmadi

ME 529 - Stochastics

For Continuous Random Process For Continuous Random Process For Markov Processes For Markov Processes

( ) ( ) ( ) ( )

∫ ∫

∞ + ∞ − +∞ ∞ −

= =

1 1 1 1 1 1 1 1

dx t , x | t , x f t , x ; t , x | t , x f dx t , x | t , x ; t , x f t , x | t , x f

( ) ( )

1 1 1 1

, | , , ; , | , t x t x f t x t x t x f =

( ) ( ) ( )

∫

+∞ ∞ −

=

1 1 1 1 1

, | , , | , , | , dx t x t x f t x t x f t x t x f

• G. Ahmadi

ME 529 - Stochastics

( ) [ ] ( ) [ ]

f t x x f t x x t f , 2 1 ,

11 2 2 1

α α ∂ ∂ + ∂ ∂ − = ∂ ∂

( ) ( ) ( ) { }

x t X t dX E dt t x

dt

= =

→

| 1 lim ,

1

α

( ) ( ) [ ] ( )

{ }

x t X t dX E dt t x

dt

= =

→

| 1 lim ,

2 11

α

( ) ( )

2 , ,

2 2 11 1

= ∂ ∂ + ∂ ∂ + ∂ ∂ x f t x x f t x t f α α

( )

0,

| , t x t x f f =

Kolmogorov Kolmogorov Equation Equation

SLIDE 3

3

• G. Ahmadi

ME 529 - Stochastics

Recall Recall Let Let

( ) { }

= t W E

( )

{ }

Dt t W E 2

2

=

( )

⎭ ⎬ ⎫ ⎩ ⎨ ⎧ ≥ ≥ =

2 1 2 1 2 1 2 1

2 2 , t t Dt t t Dt t t RWW

( ) ( ) { }

1 2

= − t W t W E

( ) ( ) [ ]

{ }

( )

1 2 2 1 2

2 t t D t W t W E − = −

1 2

t t > dt t t + =

2

t t =

1

( ) ( )

t W dt t W dW − + =

{ }

= dW E

( )

{ }

Ddt dW E 2

2 =

{ }

| 1 lim

1

= =

→

W dW E dt

dt

α

( )

{ }

D W dW E dt

dt

2 | 1 lim

2 11

= =

→

α

• G. Ahmadi

ME 529 - Stochastics

Corresponding Fokker Corresponding Fokker-

• Planck Equation

Planck Equation ( ) ( )

2 2 11 2 2 1

2 1 x f D f x f x t f ∂ ∂ = ∂ ∂ + ∂ ∂ − = ∂ ∂ α α

For solution is For solution is

( ) ( )

• w

w t , w | t , w f − = δ

2 2

W f D t f ∂ ∂ = ∂ ∂

( ) ( )

( )

• t

t D w w

t t D e f

• −

=

− − −

π 4

4

2

• G. Ahmadi

ME 529 - Stochastics

( )

[ ]

( )

[ ]

∑ ∑∑

=

∂ ∂ ∂ + ∂ ∂ − = ∂ ∂

n j i j ij j i j j

f t x x f t x t f

1 2

, 2 1 , x x α α

( ) ( )

{ }

x X = =

→

t t dX E dt

j dt j

| 1 lim α

( ) ( ) ( )

{ }

x X = =

→

t t dX t dX E dt

j i dt ij

| 1 lim α

• G. Ahmadi

ME 529 - Stochastics

Ito’s Equation Ito’s Equation ( ) ( ) ( )

t t t d n x G X g dt X ⋅ + = , , ( ) ( ) ( )

t n t G t g dt dX

j j ij i i

∑

+ = , , X X

( ) ( ) dW

X G X g X ⋅ + = t dt t d , ,

{ } { }

= =

i i

dW E n E

( ) ( )

{ }

( )

τ δ τ

ij j i

D t n t n E 2 = +

{ }

dt D dW dW E

ij j i

2 =

• r
• r

Here n and W being vector white noise and Here n and W being vector white noise and Wiener processes with Wiener processes with

SLIDE 4

4

• G. Ahmadi

ME 529 - Stochastics

Fokker Fokker-

• Planck Equation for Ito’s Equation

Planck Equation for Ito’s Equation

{ }

( )

t x g dX E dt a

j j dt j

, | 1 lim = = =

→

x X

{ }

x X = =

→

| 1 lim

j i dt ij

dX dX E dt a

( )

∑∑ ∑ ∑

+ + + =

k k j ik k k ik j k k jk i j i j i

dW dW G G dt dW G g dt dW G g dt g g dX dX

l l l 2

( )ij

T k k j ik ij

D G G G D G ⋅ ⋅ = = ∑∑ 2 2

l l l

α

( )

[ ] ( ) [ ]

∑∑ ∑

⋅ ⋅ ∂ ∂ ∂ + ∂ ∂ − = ∂ ∂

i j ij T j i j j j

f x x f t g x t f G D G x

2

,

• G. Ahmadi

ME 529 - Stochastics

Consider Consider Stationary Stationary Let Let

( ) ( ) dt

dW t x G t x g dt dx , , + =

( )

{ }

Ddt dW E 2

2 =

( )

( )

f G x D gf x t f

2 2 2

∂ ∂ + ∂ ∂ − = ∂ ∂

( )

2

= ⎥ ⎦ ⎤ ⎢ ⎣ ⎡ + − f G dx d D gf dx d

( )

1 2

= = + − c f G dx d D gf F f G =

2

F G g dx dF D

2

= dx DG g F dF

2

=

( ) ( )

⎭ ⎬ ⎫ ⎩ ⎨ ⎧+ =

∫

x

dx x DG x g C F

1 1 2 1

exp

( ) ( )

⎭ ⎬ ⎫ ⎩ ⎨ ⎧ =

∫

x

dx x DG x g G c f

1 1 2 1 2 exp

Fokker Fokker-

• Planck

Planck

• G. Ahmadi

ME 529 - Stochastics

Concluding Remarks Concluding Remarks

• Markov Processes

Markov Processes

• Important Properties

Important Properties

• Chapman

Chapman-

• Kolmogorov

Kolmogorov-

• Equation

Equation

• Fokker

Fokker-

• Planck Equation

Planck Equation

• Fokker

Fokker-

• Planck Equation for Vector

Planck Equation for Vector Markov Processes Markov Processes

• Fokker

Fokker-

• Planck Equation for Ito’s

Planck Equation for Ito’s Equation Equation

• G. Ahmadi

ME 529 - Stochastics