[PPT] - Stochastic optimal control problems in Banach spaces Federica PowerPoint Presentation

SLIDE 1

Stochastic optimal control problems in Banach spaces

Federica Masiero Universit` a Milano Bicocca La Londe 9-14 September 2007

SLIDE 2

PLAN

1. SDEs in Banach spaces;
2. The forward-backward system;
3. Identification of Z;
4. The optimal control problem;
5. The Hamilton Jacobi Bellman equation;
6. The case of arbitrarly growing coefficents;
7. Bibliographycal comments;
8. Application to nonlinear stochastic heat equations;
9. Application to stochastic delay equations.

1

SLIDE 3

SDEs in Banach spaces Our framework SDE with values in E ⊂ H, E Banach, H Hilbert

dXτ = [AXτ + F (Xτ)] dτ + GdWτ,

τ ∈ [t, T] Xt = x, 0 ≤ t ≤ T. Theorem 1: If Hypothesis 1 is satisfied, there exists a unique mild solution X (τ, t, x), that is an adapted and continuous E-valued process satisfying P-a.s. Xτ = e(τ−t)Ax +

τ

t

e(τ−s)AF (Xs) ds +

τ

t

e(τ−s)AGdWs, τ ∈ [t, T] . Proof: See e.g. Da Prato and Zabczyk (1992, 1996).

2

SLIDE 4

SDEs in Banach spaces Hypothesis 1

1. A generates a C0 semigroup etA, t ≥ 0, in E, and there exists ω ∈ R such

that

etA
L(E,E) ≤ eωt, for all t ≥ 0. etA, t ≥ 0 extends to a C0 semigroup
f bounded linear operators in H.
2. F : E → E continuous and ∃ η ≥ 0 s.t. A + F − ηI is dissipative in E.
3. G ∈ L (Ξ, H) and Qσ =

σ

esAGG∗esA∗ds is a trace class operator in H.

4. W cylindrical Wiener process in Ξ and WA (τ) = τ

t e(τ−s)AGdWs admits

an E-continuous version.

3

SLIDE 5

SDEs in Banach spaces Regularity with respect to the initial datum Let Hp ([0, T] , E) = predictable processes: E supτ∈[0,T] |Xτ|p

E < ∞

.

X (τ, t, x) is Lipschitz in x uniformly with respect to τ :

X (τ, t, x1) − X (τ, t, x2)E ≤ e|η|T x1 − x2E

If F is Gateaux differentiable in E, X (τ, t, ·) is pathwise differentiable.
Assume that ∃ k ≥ 0 s.t. F (x)E ≤ c
1 + xk

E

⇒
1. (X (τ, t, x))τ∈[0,T] ∈ Hp ([0, T] , E)
2. the map x → (X (τ, t, x))τ∈[0,T] from E to Hp ([0, T] , E) is Gateaux differ-

entiable.

4

SLIDE 6

The forward-backward system Forward-backward system

    

dXτ = AXτdτ + F (Xτ) dτ + GdWτ, τ ∈ [t, T] dYτ = −ψ (τ, Xτ, Zτ) dτ + ZτdWτ, τ ∈ [t, T] Xt = x, YT = φ (XT) . Hypothesis 2: • For every σ ∈ [0, T], x ∈ E and z1, z2 ∈ Ξ∗ |ψ (σ, x, z1) − ψ (σ, x, z2)| ≤ L |z1 − z2|Ξ∗ .

For every σ ∈ [0, T] ψ (σ, ·, ·) ∈ G1,1 (E × Ξ∗) and for every σ ∈ [0, T], x, h ∈ E

and z ∈ Ξ∗ |∇xψ (σ, x, z) h| ≤ L hE

1 + xE

m

1 + |z|Ξ∗

.
φ ∈ G1 (E) and lipschitz continuous on E.
F ∈ G1 (E) and ∃ k ≥ 0 s.t. F (x)E ≤ c
1 + xk

E

.

5

SLIDE 7

The forward-backward system Proposition Let hypotheses 1 and 2 hold true. Then the BSDE admits a unique solution (Y, Z) ∈ Kcont ([0, T]) and the map (t, x) → (Y (·, t, x) , Z (·, t, x)) belongs to G0,1 ([0, T] × E, Kcont ([0, T])). The following estimates holds true: for every p ≥ 2,

E sup

τ∈[0,T]

|∇xY (τ, t, x) h|p

1/p

≤ C hE

1 + x(m+1)

2

E

.

Corollary Let hypotheses 1 and 2 hold true. Then the function v (t, x) := Y (t, t, x) belongs to G0,1 ([0, T] × E, R) and there exists C > 0 such that |∇xv (t, x) h| ≤ C hE

1 + x(m+1)

2

E

for all t ∈ [0, T], x, h ∈ E.

6

SLIDE 8

Identification of Z Theorem 2 Let hypotheses 1 and 2 hold true and set v (t, x) := Y (t, t, x), Then, for almost every s ∈ [0, T], Zsξ = ∇v (s, Xs) Gξ, P-almost everywhere and for every ξ ∈ Ξ0. Main technical result. The argument generalizes the one in Bismut, Martin- gales, the Malliavin calculus and hypoellipticity under general H¨

rmander’s
conditions. Z. Wahrsch. Verw. Gebiete (1981).

More in general it holds true for v ∈ G0,1 ([0, T] × E, R) satisfying v (τ, Xτ) = v (T, XT) +

T

τ

ψσdσ −

T

τ

ZσdWσ, τ ∈ [t, T] .

7

SLIDE 9

Identification of Z Hypothesis Assume there exists a Banach subspace Ξ0 dense in Ξ s.t. G (Ξ0) ⊂ E and G : Ξ0 − → E is continuous. Theorem 2: Let X be solution of the SDE, Z and ψ be square integrable

processes. Let v ∈ G0,1 ([0, T] × E) s.t. for every 0 ≤ t ≤ s ≤ T, |∇v (s, x) h| ≤

c

1 + xj

E

hE, for some integer j ≥ 0 and for every x, h ∈ E. If

v (t, x) +

T

t

ZσdWσ = v (T, XT) +

T

t

ψσdσ, then, for almost every s ∈ [0, T],Zsξ = ∇v (s, Xs) Gξ, P-almost everywhere and for every ξ ∈ Ξ0. Remark Since Ξ0 is dense in Ξ, for every ¯ ξ ∈ Ξ there exists a sequence (ξn)n ∈ Ξ0 such that ξn − → ¯ ξ in Ξ. For almost every s ∈ [0, T] and almost surely with respect to the law of Xs, the operator ∇v (s, x) G : Ξ0 − → E extends to an operator defined in the whole Ξ. So Zs = ∇v (s, Xs) G, P-almost surely and for almost every s ∈ [0, T] .

8

SLIDE 10

Identification of Z Proof: Let η be a bounded and predictable process with the following form: let

kT

2n , (k + 1) T 2n

, k = 0, ....2n − 1 be a partition [0, T]. For t ∈

kT

2n , (k + 1) T 2n

ηt = ηk

Wt1, ..., Wtlk

, t ∈

kT

2n , (k + 1) T 2n

, 0 ≤ t1 ≤ ... ≤ tlk ≤ kT

2n , ηk ∈ C∞

b (Rlk, R).

For ς ∈ Ξ0, set ξt = ηtς. Notation: ξt = ξt (W·), where (W·) is the trajectory

f W up to time t. (see Bismut).

9

SLIDE 11

Identification of Z v (s, X (s, t, x)) +

T

s

ZσdWσ = v (T, XT) +

T

s

ψσdσ and ,for t ≤ s ≤ T, v (s, X (s, t, x)) = v (t, x) +

s

t

ZσdWσ −

s

t

ψσdσ.

E

v (s, Xs)

s

s−δ

ξ∗

σdWσ

= −E

s−δ

t

ψσdσ

s

s−δ

ξ∗

σdWσ

− E

s

s−δ

ψσdσ

s

s−δ

ξ∗

σdWσ

+ E

s

t

ZσdWσ

s

s−δ

ξ∗

σdWσ

.

⇓

E [Zsξs] = lim

δ→0

1 δ E

v (s, Xs)

s

s−δ

ξ∗

σdWσ

.

10

SLIDE 12

Identification of Z Prove that lim

δ→0

1 δ E

v (s, Xs)

s

s−δ

ξ∗

σdWσ

= E [∇v (s, Xs) Gξs] .

Following Bismut, W ε

σ = Wσ − ε

σ

t

ξr (W ε

· ) dr,

W ε

σ = W ε σ (W·). So

W ε

σ = Wσ − ε

σ

t

ξr (W ε

· (W·)) dr, 0 ≤ t ≤ σ ≤ T.

and d dε|ε=0W ε

σ =

σ

t

ξr (W·) dr.

11

SLIDE 13

Identification of Z Let dQε dP = exp

ε

T

t

ξ∗

σ (W ε · (W·)) dWσ − ε2

2 T

t

|ξσ (W ε

· (W·))|2 dσ

By dominated convergence

E

v (s, Xs)

s

t

ξ∗

σdWσ

= d

dε|ε=0E

v (s, Xs) exp
ε

s

t

ξ∗

σdWσ − ε2

2 s

t

|ξσ|2 dσ

= d

dε|ε=0EQε [v (s, Xs)] .

12

SLIDE 14

Identification of Z Under Qε X solves

dXτ = AXτdτ + F (Xτ) dτ + Gεξτdτ + GdW ε

τ ,

τ ∈ [s − δ, T] Xs−δ = X (s − δ, t, x) . Under P Xε solves

dXε

τ = AXε τdτ + F (Xε τ) dτ + Gεξτdτ + GdWτ,

τ ∈ [s − δ, T] Xε

t = X (s − δ, t, x) .

⇓ d dε|ε=0EQε [v (s, Xs)] = d dε|ε=0E [v (s, Xε

s)] = E

∇v (s, Xs)

·

Xs

13

SLIDE 15

Identification of Z

d

·

Xτ = A

·

Xτdτ + ∇F (Xτ)

·

Xτdτ + Gξτdτ, τ ∈ [s − δ, T]

·

Xs−δ = 0. claim:

·

Xτ = τ

t ∇X (τ, σ, X (σ, t, x)) Gξσdσ

⇓

E [Zsξs] = lim

δ→0

1 δ E

∇v (s, Xs)

s

s−δ

ξ∗

σdWσ

= lim

δ→0

1 δ E

∇v (s, Xs)

s

s−δ

∇X (s, σ, X (σ, t, x)) Gξσdσ

= E [∇v (s, Xs) ∇X (s, s, X (s, t, x)) Gξs]

= E [∇v (s, Xs) Gξs] .

14

SLIDE 16

The optimal control problem The optimal control problem: weak formulation controlled SDE

dXu

τ = [AXu τ + F (Xu τ ) + GR (τ, Xu τ , uτ)] dτ + GdWτ

Xu

t = x ∈ E,

τ ∈ [t, T] . u ∈ L2

P (Ω × [0, T] , U) .

Cost functional and value function

A = (Ω, F, Fτ, P, W, u, Xu) admissible control system (a.c.s.).

J (t, x, A) = E

T

t

g (s, Xu

s , us) ds + Eφ (Xu T) ,

J∗ (t, x) = inf

A J (t, x, A) .

15

SLIDE 17

The optimal control problem Hypothesis 3: R : [0, T] × H × U − → Ξ measurable. ∀τ ∈ [0, T] , |R (τ, x, u)| ≤ KR. φ ∈ G1 (E) and lipschitz continuous on E. g : [0, T] × E × U − → R, continuous. ∃ K > 0 s.t. for j ≥ 0, for every x ∈ E |g (τ, x, u)| ≤ K

1 + xj

E

.

Weak formulation of the optimal control problem (see e.g. Fleming-Soner 1993): find an a.c.s. A s.t. J

t, x, A
≤ J (t, x, A) for every a.c.s. A. Then A

is optimal.

16

SLIDE 18

The optimal control problem Hamiltonian function: for every τ ∈ [0, T] , x ∈ E, z ∈ Ξ∗, ψ (τ, x, z) = inf {g (τ, x, u) + zR (τ, x, u) : u ∈ U} . We assume that ψ (σ, ·, ·) ∈ G1,1 (E × Ξ∗) and |∇xψ (σ, x, z) h| ≤ L hE

1 + xE

m

1 + |z|Ξ∗

.

Fundamental relation in terms of BSDE v (t, x) = J (t, x, A) + E

T

t

[ψ (σ, Xu

σ, Zu σ) − Zu σR (σ, Xu σ, uσ) − g (σ, Xu σ, uσ)] dσ.

Moreover Zu

σ = ∇v (σ, Xu σ) G.

17

SLIDE 19

The optimal control problem u is optimal iff uτ ∈ Γ (τ, Xu

τ , ∇v (τ, Xu τ ) G) ,

P-a.s. for a.a. τ ∈ [t, T] .

Closed loop equation: for τ ∈ [t, T]

dXτ =
AXτ + F
Xτ
+ GR
τ, Xτ, Γ
τ, Xτ, ∇v
τ, Xτ
G
dτ + GdWτ,

Xt = x. Theorem 3: Under hypotheses 1, 2, 3 there exists an optimal a.c.s. and the

ptimal trajectory is solution of the closed loop equation.

18

SLIDE 20

The Hamilton Jacobi Bellman equation Hamilton Jacobi Bellman equation on a Banach space E.

∂v

∂t(t, x) = −Av (t, x) − ψ (t, x, ∇v (t, x) G) ,

t ∈ [0, T] , x ∈ E v(T, x) = φ (x) , where Af (x) = 1 2TraceH

GG∗∇2f (x)
+ Ax, ∇f (x)E,E∗ + F (x) , ∇f (x)E,E∗.

Let Pt,τ [φ] (x) = Eφ (X (τ, t, x)). v mild solution v(t, x) = Pt,T [φ] (x) +

T

t

Pt,τ [ψ(τ, ·, ∇v (τ, ·) G] (x) dτ, t ∈ [0, T] , x ∈ E. Theorem 4: With hypothesis 1 and 2, there exists a unique mild solution v(t, x) = Y (t, t, x), where (Y, Z) is the unique solution of the BSDE.

19

SLIDE 21

The case of arbitrarly growing coefficents

dXτ = [AXτ + F (Xτ)] dτ + GdWτ,

τ ∈ [t, T] Xt = x, 0 ≤ t ≤ T. Do not impose any growth conditions on F. If F is Gateaux differentiable in E, X (τ, t, ·) is pathwise differentiable. In general, X / ∈ Hp ([0, T] , E). ψ and φ in the BSDE

  

dYτ = −ψ (τ, Xτ, Zτ) dτ + ZτdWτ, τ ∈ [t, T] YT = φ (XT) . must be taken bounded.

Recover differentiability of Y and Z w.r.to x.
Identification of Z.

20

SLIDE 22

Bibliographycal comments

HJB equations , stochastic optimal control: the BSDE approach

Pardoux, Peng, Backward stochastic differential equations and quasi- linear parabolic partial differential equations, 1992. Fuhrman, Tessitore, Non linear Kolmogorov equations in infinite dimen- sional spaces: the backward stochastic differential equations approach and applications to optimal control, 2002.

HJB equations and stochastic optimal control in infinite dimensions

Da Prato, Zabczyk, Second order partial differential equations in Hilbert spaces, 2002. Gozzi, Regularity of solutions of second order Hamilton-Jacobi equations in Hilbert spaces and applications to a control problem, 1995. Cannarsa, Da Prato, Second order Hamilton-Jacobi equations in infinite dimensions, 1991, Direct solution of a second order Hamilton-Jacobi equations in Hilbert spaces, 1992.

21

SLIDE 23

Bibliographycal comments

Viscosity solutions of infinite dimensional HJB equations

Crandall, Ishii, Lions, User’s guide to viscosity solutions of second order partial differential equations, 1992. Crandall, Lions, Viscosity solutions of Hamilton-Jacobi equations in Ba- nach spaces, 1984. Crandall, Lions, Solutions de viscosit´ e pour les ´ equations de Hamilton- Jacobi dans des espaces de Banach, 1985.

The Banach space case

M., Regularizing properties for transition semigroups and semilinear par- abolic equations in Banach spaces, 2007

22

SLIDE 24

Application to nonlinear stochastic heat equations Stochastic controlled semilinear heat equations on [0, 1].

    

dτXu (τ, ξ) =

∂2

∂ξ2Xu (τ, ξ) + f (τ, ξ, Xu (τ, ξ)) + χO(ξ)u (τ, ξ)

dτ + χO (ξ) ˙

W (τ, ξ) dτ, Xu (τ, 0) = Xu (τ, 1) = 0, Xu (t, ξ) = x0 (ξ) , O ⊂ [0, 1], namely O = [a, b]. cost functional J (t, x, (W, U, xu)) = E

T

t

1 l (s, ξ, Xu (s, ξ) , u) µ (dξ) ds+E

1 k (ξ, Xu (T, ξ)) µ (dξ)

23

SLIDE 25

Application to nonlinear stochastic heat equations Abstract formulation H = Ξ = L2 ([0, 1]) , E = C ([0, 1]).

dXu

τ = [AXu τ + F (τ, Xu τ )] dτ + GRuτdτ + GdWτ

τ ∈ [t, T] Xu

t = x0,

where F (τ, x) (ξ) = f (τ, ξ, x (ξ)) , g (τ, x, u) (ξ) =

1 l (s, ξ, x (ξ) , u) µ (dξ) , (Gz) (ξ) = χO (ξ) z (ξ) φ (x) (ξ) =

1 k (ξ, x (ξ)) µ (dξ) .

24

SLIDE 26

Application to nonlinear stochastic heat equations Hypothesis

f ∈ C([0, T] × [0, 1] × R, R), ∀ τ ∈ [0, T], ξ ∈ [0, 1], f (τ, ξ, ·) ∈ C1 (R). ∀

τ ∈ [0, T] , ξ ∈ [0, 1] , x ∈ R the map x − → f(τ, ξ, x) is decreasing

l ∈ Cb([0, T] × [0, 1] × R × U, R).
k ∈ Cb([0, 1] × R, R) and k (ξ, ·) ∈ C1

b (R).

x0 ∈ C ([0, 1]).

Ξ0 = {f ∈ C ([0, 1]) : f(a) = f(b) = 0}, where O = [a, b]. So G : Ξ0 − → E = C([0, 1]).

25

SLIDE 27

Application to nonlinear stochastic heat equations An example of a cost: µ = N

i=1 δξi, ξ1, ..., ξN ∈ [0, 1] .

Hamiltonian ψ (t, x, z) = inf

u∈U

N

i=1

l (t, ξi, x (ξi) , u) +

1 z (ξ) r (ξ) u (ξ) dξ

.

Let ψ : [0, T] × RN × Ξ∗ → R ψ (t, y1, ..., yN, z) = inf

u∈U

N

i=1

l (t, ξi, yi, u) +

1 z (ξ) r (ξ) u (ξ) dξ

.

If N

i=1 l (t, ξi, yi, u) = N i=1 l (t, ξi, yi) +

1 u2 (ξ) 2 dξ, then ψ (t, ·, ..., ·, ·) : RN × Ξ∗ → R is differentiable with bounded derivatives.

26

SLIDE 28

Application to nonlinear stochastic heat equations Lemma: WA (τ) admits a continuous version in C ([0, T] , E). Theorem: Under the previous assumptions, equation

∂v

∂t(t, x) = −Atv (t, x) − ψ (t, x, ∇v (t, x) G) ,

t ∈ [0, T] , x ∈ H, u(T, x) = φ (x) , has a unique mild solution v and for all admissible control systems (W, u, Xu), J (t, x, (W, u, Xu)) ≥ v (t, x). Moreover there exists an optimal a.c.s. and the

ptimal trajectory is solution of the closed loop equation.

27

SLIDE 29

Application to stochastic delay equations Stochastic delay equations

      

dzu (τ) =

−r

dη (θ) zu (τ + θ)

dτ + u (τ) dτ + dW (τ) ,

τ ∈ [0, T] zu (0) = h0 ∈ R, zu (θ) = h1 (θ) , θ ∈ [−r, 0] , h1 ∈ Lp ([−r, 0] , R) . Cost functional J (t, h0, h1, u) = E

−r

zu (T + θ) y (θ) dθ, where y ∈ Lq ([−r, 0] , R).

28

SLIDE 30

Application to stochastic delay equations Define E = R ⊕ Lp ([−r, 0] , R) and A by D (A) =

h0

h1

∈ E, h1 ∈ W 1,p ([−r, 0] , R) , h1 (0) = h0
,

Ah = A

h0

h1

=

 

−r

a (dθ) h1 (θ) dh1/dθ

  .

Abstract formulation in E: Xτ =

z (τ)

zτ

, where zτ (θ) = z (τ + θ).
dXτ = AXτdτ + Guτdτ + GdWτ,

τ ∈ [0, T] X0 = h. where G : R − → E, G =

I
.

29

SLIDE 31

Application to stochastic delay equations Set for j =

j1

j2

∈ E, φ (j) =

−r

j2 (θ) y (θ) dθ. Abstract cost functional J (t, h, u) = Eφ (Xu

T) .

Theorem 5: There exists an optimal a.c.s. and the optimal trajectory is solution of the closed loop equation.

30

SLIDE 32

Application to stochastic delay equations Application to finance

dSτ = rSτdWτ,

τ ∈ [0, T] S0 = s0. exotic option with contingent claim ϕ (ST (·)) , where ST (θ) = ST+θ, θ ∈ [−T, 0] . We can treat contingent claims such as ϕ (ST (·)) =

−T

k (θ) ST (θ) dθ,

btaining an infinite dimensional analogue of Black-Scholes

31