[PPT] - Generalized finite differences for solving stochastic control PowerPoint Presentation

SLIDE 1

Generalized finite differences for solving stochastic control problems

March 2005

F . Bonnans, INRIA Rocquencourt

H. Zidani, ENSTA and INRIA

Contributions from our Ph.D. students: E. Ottenwaelter and S. Maroso http://www-rocq.inria.fr/sydoco

Generalized finite differencesfor solving stochastic control problems – p.1/37

SLIDE 2

Stochastic optimal control problem

(Px)                        Min I E ∞ ℓ(y(t), u(t))e−λtdt;

dy(t) = f(y(t), u(t))dt + σ(y(t), u(t))dw(t),

y(0) = x, u(t) ∈ U, t ∈ [0, ∞[.

Generalized finite differencesfor solving stochastic control problems – p.2/37

SLIDE 3

Notation

y(t) ∈ I

Rn : state variable,

Generalized finite differencesfor solving stochastic control problems – p.3/37

SLIDE 4

Notation

y(t) ∈ I

Rn : state variable,

u(t) ∈ I

Rm : control variable,

Generalized finite differencesfor solving stochastic control problems – p.3/37

SLIDE 5

Notation

y(t) ∈ I

Rn : state variable,

u(t) ∈ I

Rm : control variable,

λ ≥ 0 : discounting factor,

Generalized finite differencesfor solving stochastic control problems – p.3/37

SLIDE 6

Notation

y(t) ∈ I

Rn : state variable,

u(t) ∈ I

Rm : control variable,

λ ≥ 0 : discounting factor,
ℓ : I

Rn × I Rm → I R : distributed cost,

Generalized finite differencesfor solving stochastic control problems – p.3/37

SLIDE 7

Notation

y(t) ∈ I

Rn : state variable,

u(t) ∈ I

Rm : control variable,

λ ≥ 0 : discounting factor,
ℓ : I

Rn × I Rm → I R : distributed cost,

f : I

Rn × I Rm → I Rn : deterministic dynamics,

Generalized finite differencesfor solving stochastic control problems – p.3/37

SLIDE 8

Notation

y(t) ∈ I

Rn : state variable,

u(t) ∈ I

Rm : control variable,

λ ≥ 0 : discounting factor,
ℓ : I

Rn × I Rm → I R : distributed cost,

f : I

Rn × I Rm → I Rn : deterministic dynamics,

σ(·) : I

Rn × I Rm → space of n × r matrices

Generalized finite differencesfor solving stochastic control problems – p.3/37

SLIDE 9

Notation

y(t) ∈ I

Rn : state variable,

u(t) ∈ I

Rm : control variable,

λ ≥ 0 : discounting factor,
ℓ : I

Rn × I Rm → I R : distributed cost,

f : I

Rn × I Rm → I Rn : deterministic dynamics,

σ(·) : I

Rn × I Rm → space of n × r matrices

w :standard r dimensional Brownian motion.

Generalized finite differencesfor solving stochastic control problems – p.3/37

SLIDE 10

Notation

y(t) ∈ I

Rn : state variable,

u(t) ∈ I

Rm : control variable,

λ ≥ 0 : discounting factor,
ℓ : I

Rn × I Rm → I R : distributed cost,

f : I

Rn × I Rm → I Rn : deterministic dynamics,

σ(·) : I

Rn × I Rm → space of n × r matrices

w :standard r dimensional Brownian motion.
Value function V of problem (Px): finite and

continuous

Generalized finite differencesfor solving stochastic control problems – p.3/37

SLIDE 11

HJB equation

λv(x) = inf

u∈U{ℓ(x, u) + f(x, u) · vx(x)

+1

2

n

i,j=1 aij(x, u) vxixj(x)},

for all x ∈ I Rn.

Generalized finite differencesfor solving stochastic control problems – p.4/37

SLIDE 12

HJB equation

λv(x) = inf

u∈U{ℓ(x, u) + f(x, u) · vx(x)

+1

2

n

i,j=1 aij(x, u) vxixj(x)},

for all x ∈ I Rn.

Covariance matrix: a(x, u) :=

σ(x, u)σ(x, u)T, ∀ (x, u) ∈ I Rn × I Rm.

Generalized finite differencesfor solving stochastic control problems – p.4/37

SLIDE 13

HJB equation

λv(x) = inf

u∈U{ℓ(x, u) + f(x, u) · vx(x)

+1

2

n

i,j=1 aij(x, u) vxixj(x)},

for all x ∈ I Rn.

Covariance matrix: a(x, u) :=

σ(x, u)σ(x, u)T, ∀ (x, u) ∈ I Rn × I Rm.

All functions Lipschitz continuous and

bounded: V unique bounded viscosity solution of HJB.

Generalized finite differencesfor solving stochastic control problems – p.4/37

SLIDE 14

Simplified HJB equation

Discretization already non-trivial in this case: Null drift f and data not depending on control variable u. Then the HJB equation reduces to: λv(x) = ℓ(x) + 1

2 n

i,j=1

aij(x) vxixj(x), for all x ∈ I Rn

Generalized finite differencesfor solving stochastic control problems – p.5/37

SLIDE 15

Generalized finite differences I

p O ξ2 ξ1

Generalized finite differencesfor solving stochastic control problems – p.6/37

SLIDE 16

Generalized finite differences II

Discretization steps h1, . . . , hn, points

xk := (k1h1, . . . , knhn).

Generalized finite differencesfor solving stochastic control problems – p.7/37

SLIDE 17

Generalized finite differences II

Discretization steps h1, . . . , hn, points

xk := (k1h1, . . . , knhn).

Given ϕ = {ϕk}: real valued function over Zn.

Generalized finite differencesfor solving stochastic control problems – p.7/37

SLIDE 18

Generalized finite differences II

Discretization steps h1, . . . , hn, points

xk := (k1h1, . . . , knhn).

Given ϕ = {ϕk}: real valued function over Zn.
Second order finite difference operator

∆ξϕk := ϕk+ξ+ϕk−ξ−2ϕk = ϕk+ξ−ϕk−(ϕk−ϕk−ξ).

Generalized finite differencesfor solving stochastic control problems – p.7/37

SLIDE 19

Generalized finite differences II

Discretization steps h1, . . . , hn, points

xk := (k1h1, . . . , knhn).

Given ϕ = {ϕk}: real valued function over Zn.
Second order finite difference operator

∆ξϕk := ϕk+ξ+ϕk−ξ−2ϕk = ϕk+ξ−ϕk−(ϕk−ϕk−ξ).

If ϕk = Φ(xk). Then

∆ξϕk = D2Φ(xk)xξxξ + o(xξ2) = n

i,j=1 ξihiξjhjΦxixj(xk) + o(h2)

Generalized finite differencesfor solving stochastic control problems – p.7/37

SLIDE 20

Approximation of second-order term

Scaled covariance: ah

ij(x) = a(x)/(hihj).

Generalized finite differencesfor solving stochastic control problems – p.8/37

SLIDE 21

Approximation of second-order term

Scaled covariance: ah

ij(x) = a(x)/(hihj).

Strong consistency:
ξ∈S αk,ξ∆ξφk = n

i,j=1 aij(xk)Φxixj(xk)

Generalized finite differencesfor solving stochastic control problems – p.8/37

SLIDE 22

Approximation of second-order term

Scaled covariance: ah

ij(x) = a(x)/(hihj).

Strong consistency:
ξ∈S αk,ξ∆ξφk = n

i,j=1 aij(xk)Φxixj(xk)

Characterization:
ξ∈S αk,ξξξT = ah(xk),

for all k ∈ Zn.

Generalized finite differencesfor solving stochastic control problems – p.8/37

SLIDE 23

Numerical scheme

Explicit schemes:

λvk = ℓ(xk) +

ξ∈S αk,ξ∆ξvk, k ∈ Zn

Generalized finite differencesfor solving stochastic control problems – p.9/37

SLIDE 24

Numerical scheme

Explicit schemes:

λvk = ℓ(xk) +

ξ∈S αk,ξ∆ξvk, k ∈ Zn

Fictitious time step: h0 > 0. Equivalent

scheme: vk := (1+λh0)−1   vk + h0ℓ(xk) + h0

ξ∈S

αk,ξ∆ξvk   

Generalized finite differencesfor solving stochastic control problems – p.9/37

SLIDE 25

Monotonicity condition

Nondecreasing mapping

v → vk + h0ℓ(xk) + h0

ξ∈S αk,ξ∆ξvk

Generalized finite differencesfor solving stochastic control problems – p.10/37

SLIDE 26

Monotonicity condition

Nondecreasing mapping

v → vk + h0ℓ(xk) + h0

ξ∈S αk,ξ∆ξvk
Holds iff

αk,ξ ≥ 0, ∀ (ξ, k) ∈ S × Zn. 2

ξ∈S

αk,ξ ≤ h−1

0 , ∀ (k) ∈ Zn.

Generalized finite differencesfor solving stochastic control problems – p.10/37

SLIDE 27

Monotonicity condition

Nondecreasing mapping

v → vk + h0ℓ(xk) + h0

ξ∈S αk,ξ∆ξvk
Holds iff

αk,ξ ≥ 0, ∀ (ξ, k) ∈ S × Zn. 2

ξ∈S

αk,ξ ≤ h−1

0 , ∀ (k) ∈ Zn.

Second condition satisfied when h0 small

enough, once an estimate of

ξ∈S αk,ξ is

known (see below).

Generalized finite differencesfor solving stochastic control problems – p.10/37

SLIDE 28

Convergence

Monotonicity and consistency imply

convergence.

Generalized finite differencesfor solving stochastic control problems – p.11/37

SLIDE 29

Convergence

Monotonicity and consistency imply

convergence.

Strong consistency implies
ξ∈S αk,ξ ≤ trace ah(xk)

Generalized finite differencesfor solving stochastic control problems – p.11/37

SLIDE 30

Convergence

Monotonicity and consistency imply

convergence.

Strong consistency implies
ξ∈S αk,ξ ≤ trace ah(xk)
Condition for time step: h0 = O(mini h2

i)

Generalized finite differencesfor solving stochastic control problems – p.11/37

SLIDE 31

Explicit strong consistency conditions

Characterization: ah(xk) belongs to

C(S) :=

ξ∈S αξξξT; α ∈ I

R|S|

+

.

Generalized finite differencesfor solving stochastic control problems – p.12/37

SLIDE 32

Explicit strong consistency conditions

Characterization: ah(xk) belongs to

C(S) :=

ξ∈S αξξξT; α ∈ I

R|S|

+

.
Neighbours of order p:

Sp = {ξ ∈ I Rn; |ξi| ≤ p}.

Generalized finite differencesfor solving stochastic control problems – p.12/37

SLIDE 33

Explicit strong consistency conditions

Characterization: ah(xk) belongs to

C(S) :=

ξ∈S αξξξT; α ∈ I

R|S|

+

.
Neighbours of order p:

Sp = {ξ ∈ I Rn; |ξi| ≤ p}.

Invariance conditions:

(i) Permutation of coordinate (ii) Change of sign on one coordinate

Generalized finite differencesfor solving stochastic control problems – p.12/37

SLIDE 34

Explicit strong consistency conditions

Characterization: ah(xk) belongs to

C(S) :=

ξ∈S αξξξT; α ∈ I

R|S|

+

.
Neighbours of order p:

Sp = {ξ ∈ I Rn; |ξi| ≤ p}.

Invariance conditions:

(i) Permutation of coordinate (ii) Change of sign on one coordinate

Qhull algorithm by Barbet et al., absolute

precision of 10−10

Generalized finite differencesfor solving stochastic control problems – p.12/37

SLIDE 35

Some results for n = 2

C(S1) = {a; aii ≥ |aij|,

1 ≤ i = j ≤ 2}.

Generalized finite differencesfor solving stochastic control problems – p.13/37

SLIDE 36

Some results for n = 2

C(S1) = {a; aii ≥ |aij|,

1 ≤ i = j ≤ 2}.

C(S2) =
2aii ≥ |aij|

2aii + ajj ≥ 3|aij|. , 1 ≤ i = j ≤ 2

Generalized finite differencesfor solving stochastic control problems – p.13/37

SLIDE 37

Some results for n = 2

C(S1) = {a; aii ≥ |aij|,

1 ≤ i = j ≤ 2}.

C(S2) =
2aii ≥ |aij|

2aii + ajj ≥ 3|aij|. , 1 ≤ i = j ≤ 2

C(S3) =

         3aii ≥ |aij| 3aii + 2ajj ≥ 5|aij|. 6aii + ajj ≥ 5|aij|. 6aii + 2ajj ≥ 7|aij|. 1 ≤ i = j ≤ 2

Generalized finite differencesfor solving stochastic control problems – p.13/37

SLIDE 38

Some results for n = 2

C(S1) = {a; aii ≥ |aij|,

1 ≤ i = j ≤ 2}.

C(S2) =
2aii ≥ |aij|

2aii + ajj ≥ 3|aij|. , 1 ≤ i = j ≤ 2

C(S3) =

         3aii ≥ |aij| 3aii + 2ajj ≥ 5|aij|. 6aii + ajj ≥ 5|aij|. 6aii + 2ajj ≥ 7|aij|. 1 ≤ i = j ≤ 2

Generalized finite differencesfor solving stochastic control problems – p.13/37

SLIDE 39

Overview of conditions: n = 2

Size

f generator

Constraints Constraints Equiv.

f p
f primal cone

defining S∗ defining C classes

1 4 6 4 1 2 8 13 8 2 3 16 27 16 4 4 24 39 24 6 5 40 67 40 10 6 48 87 48 12 7 72 123 72 18

Generalized finite differencesfor solving stochastic control problems – p.14/37

SLIDE 40

Maximal error: n = 2

ε = sup

aF =1

a − PC(Sp)(a)F = O(p−2) p ε ε p 1 0.169102 10−1 2 2 0.055642 10−2 5 3 0.026325 10−3 16 4 0.015153 10−4 20 5 0.009804 10−5 159 15 0.001109 10−7 1 582

Generalized finite differencesfor solving stochastic control problems – p.15/37

SLIDE 41

Regular grid discretization: n = 2

η η θ

O

1 2 3

Generalized finite differencesfor solving stochastic control problems – p.16/37

SLIDE 42

Stern-Brocot tree

1 1

ց ւ

1 1

ւ ց

1 2 2 1

ւ ց ւ ց

1 3 2 3 3 2 3 1

Generalized finite differencesfor solving stochastic control problems – p.17/37

SLIDE 43

Family relations

O 1 2 3 4 5 6 7 1 2 3 4 5 6 7

Generalized finite differencesfor solving stochastic control problems – p.18/37

SLIDE 44

Covariances (n = 2)

Cone C of symmetric semidefinite positive

matrices. Extreme points:

σσT =

σ2

1

σ1σ2 σ1σ2 σ2

2

,

where σ ∈ I R2. View C as a subset of I R3:

a11 a12

a21 a22

→

   a11 √ 2a12 a22   

Generalized finite differencesfor solving stochastic control problems – p.19/37

SLIDE 45

Isometry: illustration

qII qIV qI qIII ξ2 QIV ξ1 O QIII

Ω

QII QI z2 0 z2 < 0 z1 < z3 z1 z

Generalized finite differencesfor solving stochastic control problems – p.20/37

SLIDE 46

View of SDP cone C

O Ω H13 QII QIII 1 1 H2 QIV QI D z3 z1 z2 ∂C

Cone of semidefinite positive

Generalized finite differencesfor solving stochastic control problems – p.21/37

SLIDE 47

Correspondance grid / section of cone

η η θ

O

1 2 3

a 2 θ Ω a0

Generalized finite differencesfor solving stochastic control problems – p.22/37

SLIDE 48

Intersection with trace x = 1

O 1 2 3 4 5 6 7 1 2 3 4 5 6 7

Ω

1 1 1 1 2 1 3 2 3 1 4 3 4

Generalized finite differencesfor solving stochastic control problems – p.23/37

SLIDE 49

Recursive approximations

p: size of neighbourhood (as small as

possible !)

Generalized finite differencesfor solving stochastic control problems – p.24/37

SLIDE 50

Recursive approximations

p: size of neighbourhood (as small as

possible !)

Ppa := projection of a onto Cp := C(Sp)

Generalized finite differencesfor solving stochastic control problems – p.24/37

SLIDE 51

Recursive approximations

p: size of neighbourhood (as small as

possible !)

Ppa := projection of a onto Cp := C(Sp)
Desirable feature: fast computation of Pp+1a,

having computed Ppa

Generalized finite differencesfor solving stochastic control problems – p.24/37

SLIDE 52

Recursive approximations

p: size of neighbourhood (as small as

possible !)

Ppa := projection of a onto Cp := C(Sp)
Desirable feature: fast computation of Pp+1a,

having computed Ppa

Stop when a − PpaF ≤ ε.

Generalized finite differencesfor solving stochastic control problems – p.24/37

SLIDE 53

Useful observations: n = 2

Generators of Cp are extreme directions of

SDP cone

Generalized finite differencesfor solving stochastic control problems – p.25/37

SLIDE 54

Useful observations: n = 2

Generators of Cp are extreme directions of

SDP cone

If Ppa belongs to Cp: stop

Generalized finite differencesfor solving stochastic control problems – p.25/37

SLIDE 55

Useful observations: n = 2

Generators of Cp are extreme directions of

SDP cone

If Ppa belongs to Cp: stop
Otherwise: projection on supporting

hyperplane generated by two adjacent generators

Generalized finite differencesfor solving stochastic control problems – p.25/37

SLIDE 56

Useful observations: n = 2

Generators of Cp are extreme directions of

SDP cone

If Ppa belongs to Cp: stop
Otherwise: projection on supporting

hyperplane generated by two adjacent generators

Update: add their son and check the next two

supporting hyperplanes

Generalized finite differencesfor solving stochastic control problems – p.25/37

SLIDE 57

Useful observations: n = 2

Generators of Cp are extreme directions of

SDP cone

If Ppa belongs to Cp: stop
Otherwise: projection on supporting

hyperplane generated by two adjacent generators

Update: add their son and check the next two

supporting hyperplanes

Cost of computation: O(p)

Generalized finite differencesfor solving stochastic control problems – p.25/37

SLIDE 58

Numerical inconsistency I

Test function:

     W(t, x1, x2) = (1 + t) sin(x1) sin(x2) W(0, x1, x2) = sin(x1) sin(x2) 0 x1 π; 0 x2 π; 0 t 1 ∆x := ∆x1 = ∆x2 e := Wapprox − Wexact1 N1 × N2 .

Generalized finite differencesfor solving stochastic control problems – p.26/37

SLIDE 59

Numerical inconsistency II

                           ℓ(t, x1, x2) = sin x1 sin x2[1 + α(1 + t)] −2(1 + t) cos x1 cos x2 sin(x1 + x2) cos(x1 + x2) f1(x1, x2) = f2(x1, x2) = 0 a11(x1, x2) = sin2(x1 + x2) + β a12(x1, x2) = sin(x1 + x2) cos(x1 + x2) a22(x1, x2) = cos2(x1 + x2) + β

Parameters: α = 1.2, 1, 1.2;

β = 0.1, 0, 0.

Generalized finite differencesfor solving stochastic control problems – p.27/37

SLIDE 60

Numerical inconsistency III

−3.5 −3 −2.5 −2 −1.5 −1 −10 −9 −8 −7 −6 −5 −4 −3

log ∆ x log e

Ex 1 Ex 2 Ex 3

Generalized finite differencesfor solving stochastic control problems – p.28/37

SLIDE 61

Numerical result, Optimal control

f(t, x, u) = u; u2

1 + u2 2 ≤ 1.

σ1(t, x1, x2) = √ 2 sin(x1 + x2) σ2(t, x1, x2) = √ 2 cos(x1 + x2) Expression of ℓ:

sin(x1) sin(x2) + (1 + t)

cos2(x1) sin2(x2) + sin2(x1) cos2(x2)

1/2 + sin(x1) sin(x2) − 2 sin(x1 + x2) cos(x1 + x2) cos(x1) cos(x2)]

Generalized finite differencesfor solving stochastic control problems – p.29/37

SLIDE 62

Numerical result, Optimal control

−1.6 −1.4 −1.2 −1 −0.8 −0.6 −0.4 −0.2 −2.6 −2.4 −2.2 −2 −1.8 −1.6 −1.4 −1.2

log10 ∆ x log10 e p = 4 p = 2 p = 1 p =10

Figure 0: Error vs discretization step

Generalized finite differencesfor solving stochastic control problems – p.30/37

SLIDE 63

Challenge: n = 3

Generators of Cp are generators of Cp+1

Generalized finite differencesfor solving stochastic control problems – p.31/37

SLIDE 64

Challenge: n = 3

Generators of Cp are generators of Cp+1
Relations between faces of Cp and Cp+1 ?

Generalized finite differencesfor solving stochastic control problems – p.31/37

SLIDE 65

Challenge: n = 3

Generators of Cp are generators of Cp+1
Relations between faces of Cp and Cp+1 ?
Possibility of projections on facets

Generalized finite differencesfor solving stochastic control problems – p.31/37

SLIDE 66

Splitting

We split the second member of the (HJB) equation Vt = inf

u

 Gu

0(V ) +

ξ∈S

Gu

ξ(V )

  . with Gu

0(V ) := ℓ(t, x, u) + f(t, x, u) · Vx

Gu

ξ(V ) := αξ(t, x, u) Vxx(x)(xξ, xξ) .

Generalized finite differencesfor solving stochastic control problems – p.32/37

SLIDE 67

Example in dimension 2

1 2 3 4 1 2 3

D0 D1 D2 D(N2−1)p+(N1−1)q i j

Generalized finite differencesfor solving stochastic control problems – p.33/37

SLIDE 68

Error estimates

Recent work by Krylov, Barles, Jakobsen
Best estimate for finite differences, assuming

diagonal dominent diffusion matrix: |u − uh|0 ≤ Ch1/5

Easy extension to generalized finite

differences.

Generalized finite differencesfor solving stochastic control problems – p.34/37

SLIDE 69

The adverse stopping case

Player A : She maximises the gain; Decision: control variable Player B : She minimises the gain; Decision: final time HJB equation: min{sup

u Hu(x, Dv); v − ψ} = 0

x ∈ RN where Hu(x, r, p, q) := λr − ℓ(x, u) − f(x, u) · p − 1

2

n

i,j=1 aij(x, u)qij.

Generalized finite differencesfor solving stochastic control problems – p.35/37

SLIDE 70

Error estimate for adverse stopping

Same estimate: |u − uh|0 ≤ Ch1/5 Ongoing research: extension to impulse control.

Generalized finite differencesfor solving stochastic control problems – p.36/37

SLIDE 71

Biblio

J.F

. Bonnans, S. Maroso, H. Zidani (2004). Stochastic differential games: the adverse stopping game. Rapport de Recherche INRIA RR 5441.

J.F

. Bonnans, E. Ottenwaelter, H. Zidani A fast algorithm for the 2D HJB equation of stochastic control. ESAIM:M2AN 38 (2004), 723-735.

J. F

. Bonnans and H. Zidani. Consistency of generalized finite difference schemes for the stochastic HJB equation, SIAM J. Numerical Analysis 41 (2003), 1008-1021.

Generalized finite differencesfor solving stochastic control problems – p.37/37