Numerical resolution of McKean-Vlasov Forward Backward Stochastic - - PowerPoint PPT Presentation

▶
numerical resolution of mckean vlasov forward backward
SMART_READER_LITE
LIVE PREVIEW

Numerical resolution of McKean-Vlasov Forward Backward Stochastic - - PowerPoint PPT Presentation

Oct 03, 2023 271 likes •461 views

Numerical resolution of McKean-Vlasov Forward Backward Stochastic Differential Equations using neural networks Maximilien Germain Joint work with Joseph Mikael , Xavier Warin EDF R&D, LPSM, Universit Paris Diderot 8 th January 2020 8th

slide-1
SLIDE 1

Numerical resolution of McKean-Vlasov Forward Backward Stochastic Differential Equations using neural networks

Maximilien Germain Joint work with Joseph Mikael, Xavier Warin

EDF R&D, LPSM, Université Paris Diderot

8th January 2020

Maximilien Germain ML for McKean-Vlasov FBSDEs 8th January 2020 1 / 18

slide-2
SLIDE 2

McKean-Vlasov FBSDEs

We solve general McKean-Vlasov FBSDEs of the form

  • Xt

= ξ + t

0 b(s, Xs, Ys, Zs, L(Xs), L(Ys), L(Zs)) ds +

t

0 σ(s, Xs) dWs

Yt = g(XT , L(XT )) + T

t f(s, Xs, Ys, Zs, L(Xs), L(Ys), L(Zs))ds −

T

t ZsdWs

with machine learning techniques. Ws is a standard Brownian motion. The processes dynamics depend on their laws L(·). In practise we restrict to cases when the law dependence only concerns probability moments.

Maximilien Germain ML for McKean-Vlasov FBSDEs 8th January 2020 2 / 18

slide-3
SLIDE 3

Main motivation: Mean Field Games (MFG)

Stochastic games introduced by (Lasry and Lions 2006), (Huang, Malhame, and Caines 2006) followed by (Carmona and Delarue 2012) dealing with a large number of interacting players. The empirical law of players states influences both the dynamics and cost.

Applications

Population dynamics (crowd, traffic jam, bird flocking...). Market interactions (Cardaliaguet and Lehalle 2019). Energy storage (Matoussi, Alasseur, and Ben Taher 2018), electric cars management. Social networks.

Maximilien Germain ML for McKean-Vlasov FBSDEs 8th January 2020 3 / 18

slide-4
SLIDE 4

N players stochastic game

Player i minimises a cost Ji(α1, · · · , αi, · · · , αN) depending on the empirical distribution µt = 1

N

N

k=1 δXk

t → mean field interaction.

min

αi∈A

E T f(t, Xi

t, µt, αi t) dt + g(Xi T , µT )

  • subject to

dXi

t = b(t, Xi t, µt, αi t) dt + σ(t, Xi t, µt) dW i t .

Difficult problem in general.

Maximilien Germain ML for McKean-Vlasov FBSDEs 8th January 2020 4 / 18

slide-5
SLIDE 5

Asymptotic control problem when N → +∞

Given a family (µt)t∈[0,T ] of probability measures we solve for a representative player min

α

E T f(t, Xµ

t , µt, αt) dt + g(Xµ T , µT )

  • subject to

dXµ

t = b(t, Xµ t , µt, αt) dt + σ(t, Xµ t , µt) dWt.

Fixed point on probability measures µt = L(Xµ

t ).

Simpler than the N players game → unique player in the limit! Value function given by vµ(t, x) = inf

α∈At E

T

t

f(s, Xs, µs, αs) ds + g(XT , µT )|Xt = x

  • .

Maximilien Germain ML for McKean-Vlasov FBSDEs 8th January 2020 5 / 18

slide-6
SLIDE 6

Optimality conditions

HJB equation

  • ∂tv + 1

2Tr(σσ⊤∂2 xxv) + minα∈A Hνt(t, x, ∂xv, α) = 0

v(T, x) = g(x, νT ) coupled with Fokker-Planck equation:

  • ∂tν − 1

2Tr(σσ⊤∂2 xxν) − div(b(t, x, νt, ˆ

ανt(t, x, ∂xv))ν) = 0 ν(0, ·) = ν0. Solved in (Achdou and Capuzzo-Dolcetta 2010) with finite differences methods.

Maximilien Germain ML for McKean-Vlasov FBSDEs 8th January 2020 6 / 18

slide-7
SLIDE 7

Probabilistic point of view: McKean-Vlasov FBSDEs

Yt = ∂xv(t, Xt) → stochastic Pontryagin principle          dXt = b(t, Xt, L(Xt), ˆ αL(Xt)(t, Xt, Yt)) dt + σ dWt X0 = ξ dYt = −∂xHL(Xt)(t, Xt, Yt, ˆ αL(Xt)(t, Xt, Yt)) dt + Zt dWt YT = ∂xg (XT , µT ) . In the form

  • Xt

= ξ + t

0 b(s, Xs, Ys, Zs, L(Xs), L(Ys), L(Zs)) ds +

t

0 σ(s, Xs) dWs

Yt = g(XT , L(XT )) + T

t f(s, Xs, Ys, Zs, L(Xs), L(Ys), L(Zs))ds −

T

t ZsdWs

Maximilien Germain ML for McKean-Vlasov FBSDEs 8th January 2020 7 / 18

slide-8
SLIDE 8

Advantages of machine learning techniques

Curse of dimensionnality with finite differences methods if the state dimension is high (≥ 3 or 4). − → Some machine learning schemes solve nonlinear PDE in high dimension (10, 50 or even 100): - Deep BSDE method of (Han, Jentzen, and E 2017)

  • Deep Galerkin method of (Sirignano and Spiliopoulos 2017)
  • Deep Backward Dynamic Programming of (Huré, Pham, and Warin 2020).

− → Open source libraries such as Tensorflow or Pytorch. − → Efficient computation on GPU nodes.

Maximilien Germain ML for McKean-Vlasov FBSDEs 8th January 2020 8 / 18

slide-9
SLIDE 9

A global method

Based on the Deep BSDE method of (Han, Jentzen, and E 2017)

  • Xt

= ξ + t

0 b(s, Xs, Ys, Zs, L(Xs), L(Ys), L(Zs)) ds +

t

0 σ(s, Xs) dWs

Yt = g(XT , L(XT )) + T

t f(s, Xs, Ys, Zs, L(Xs), L(Ys), L(Zs))ds −

T

t ZsdWs

Discretization:

  • Xti+1

= Xti + b (ti, Xti, Yti, Zti, µti) ∆ti + σ (ti, Xti) ∆Wti Yti+1 = Yti − f (ti, Xti, Yti, Zti, µti) ∆ti + Zθ(ti, Xti)∆Wti. Y0 is a variable and Zti is approached by a neural network Zθ(ti, ·) which minimizes the loss function E

  • (YT − g (XT , µT ))2

. The forward backward system is transformed into a forward form and an optimization problem. Also studied in (Fouque and Zhang 2019), (Carmona and Laurière 2019) in dimension 1 and with less generality.

Maximilien Germain ML for McKean-Vlasov FBSDEs 8th January 2020 9 / 18

slide-10
SLIDE 10

Fixed point on probability measures

Estimation of state and value function moments: Direct: use of current particles empirical law E[Xti](k+1) ≃ µ(k+1)

ti

= 1 Nb

Nb

  • j=1

Xj

ti.

(1) Dynamic: keep in memory previously computed moments and average them with current particle moments E[Xti](k+1) ≃ NmNbµ(k)

ti + Nb j=1 Xj ti

Nb + NmNb . (2) Expectation: estimate E[Xt] by an additional neural network. E[Xt] ≃ Ψκ(t). (3) We add to the loss a term E

  • λ Nt

i=0

  • Ψκ(ti) − Xti

2 .

Maximilien Germain ML for McKean-Vlasov FBSDEs 8th January 2020 10 / 18

slide-11
SLIDE 11

A local approach

Inspired by (Huré, Pham, and Warin 2020). Z and Y are approximated by neural networks (Zi

θi

m(·), Y i

θi

m(·)){i∈0,N−1}. At

iteration m we simulate Xi with the previously computed parameters θi

  • m. The

Xi’s dynamics being frozen with parameters θi

m:

First Y N is set to the terminal condition g (XN, µN). Then, we solve successively the local backward problems for i from N − 1 to 0

min θi m E    Y i+1 θi+1 m+1

  • Xi+1
  • − Y i

θi m

  • Xi
  • + f
  • ti, Xi, Y i

θi m

  • Xi
  • , Zi

θi m

  • Xi
  • , µti
  • ∆t − Zi

θi m

  • Xi
  • ∆Wti

  2  . Maximilien Germain ML for McKean-Vlasov FBSDEs 8th January 2020 11 / 18

slide-12
SLIDE 12

A linear quadratic test case: price impact

Large number of traders who want to sell a portfolio at the same time. Example from (Carmona and Delarue 2018). min

α∈A

E T cα 2 αt2 + cX 2 Xt2 − γXt · µt

  • dt + cg

2 XT 2

  • subject to

Xt = x0 + t αs ds + σ Wt . and the fixed point E[αt] = µt. Optimality system:          dXt = − 1

cα Yt dt + σ dWt

X0 = x0 dYt = −(cXXt + γ

cα E[Yt]) dt + Zt dWt

YT = cgXT . We take cX = 2, x0 = 1, σ = 0.7, γ = 2, cα = 2/3, cg = 0.3. The simulations are conducted with d = 10, ∆t = 0.01.

Maximilien Germain ML for McKean-Vlasov FBSDEs 8th January 2020 12 / 18

slide-13
SLIDE 13

Results 1/4: Local method/Price Impact model

0.0 0.2 0.4 0.6 0.8 1.0 −3.5 −3.0 −2.5 −2.0 −1.5 −1.0 −0.5 Optimal control Computed optimal control 1000 2000 3000 4000 5000 6000 Iteration 0.0 0.2 0.4 0.6 0.8 1.0 Mean expectation T = 0.25 T = 0.75 T = 1 T = 1.5

Figure: Computed optimal control after 6000 iterations and expectation of the state at terminal state for the local method.

Maximilien Germain ML for McKean-Vlasov FBSDEs 8th January 2020 13 / 18

slide-14
SLIDE 14

Results 2/4: Global method/Price Impact model

0.0 0.2 0.4 0.6 0.8 1.0 t −3.5 −3.0 −2.5 −2.0 −1.5 −1.0 −0.5 0.0 Optimal control Computed optimal control 250 500 750 1000 1250 1500 1750 2000 Iteration 0.0 0.5 1.0 1.5 2.0 2.5 Mean expectation T = 0.25 T = 0.75 T = 1 T = 1.5

Figure: Computed optimal control after 2000 iterations and expectation of the state at terminal state for the global method.

Maximilien Germain ML for McKean-Vlasov FBSDEs 8th January 2020 14 / 18

slide-15
SLIDE 15

A non linear quadratic test case

Take Xt log normal, Yt = exp(αt) log(

i Xi t) and quadratic dynamics.

                                                               dXi

t

= (aiXi

t + b(Yt + Zi t + E[Xi t] + E[Yt] + E[Zi t])

−b

  • eαt log

d

i=1 Xi t

  • + σi

teαt + gi t + ct + ei t

  • +c
  • Y 2

t + (Zi t)2 + E[(Xi t)2] + E[Y 2 t ] + E[(Zi t)2])

−c

  • e2αt log

d

i=1 Xi t

2 + (σi

t)2e2αt + (gi t)2 + c2 t + (ei t)2

dt + σi

tXi t dW i t

Xi = ξi dYt =

  • φ(t, Xt) + b(Yt + 1

d

d

i=1 Zi t + 1 d

d

i=1 E[Xi t] + E[Yt] + 1 d

d

i=1 E[Zi t])

−b

  • eαt log

d

i=1 Xi t

  • + 1

d

d

i=1 σi teαt + 1 d

d

i=1 gi t + ct + 1 d

d

i=1 ei t

  • +c(Y 2

t + 1 d

d

i=1(Zi t)2 + 1 d

d

i=1 E[(Xi t)2] + E[Y 2 t ] + 1 d

d

i=1 E[(Zi t)2])

−c

  • e2αt log

d

i=1 Xi t

2 + c2

t + 1 d

d

i=1(σi t)2e2αt + (gi t)2 + (ei t)2

dt + Zt dWt YT = eαT log d

i=1 Xi T

  • .

We take a = b = c = 0.1, α = 0.5, σ = 0.4, ξ = 1. The simulations are conducted with d = 10, ∆t = 0.01.

Maximilien Germain ML for McKean-Vlasov FBSDEs 8th January 2020 15 / 18

slide-16
SLIDE 16

Results 3/4: Local method/Non linear quadratic equation

0.00 0.05 0.10 0.15 0.20 0.25 0.0 0.2 0.4 0.6 Exact Y Computed Y 0.00 0.05 0.10 0.15 0.20 0.25 0.38 0.40 0.42 0.44 0.46 0.48 Exact Z Computed Z

Figure: Y and Z for the local method after 6000 iterations

Maximilien Germain ML for McKean-Vlasov FBSDEs 8th January 2020 16 / 18

slide-17
SLIDE 17

Results 4/4: Global method/Non linear quadratic equation

0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 t 0.0 0.5 1.0 1.5 2.0 Exact Y Computed Y 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 t 0.30 0.35 0.40 0.45 0.50 0.55 Exact Z Computed Z

Figure: Y and Z for the global method after 2000 iterations

Maximilien Germain ML for McKean-Vlasov FBSDEs 8th January 2020 17 / 18

slide-18
SLIDE 18

References

Yves Achdou and Italo Capuzzo-Dolcetta. “Mean Field Games: Numerical Methods”. In: SIAM Journal on Numerical Analysis 48 (Jan. 2010). doi: 10.1137/090758477. Pierre Cardaliaguet and Charles-Albert Lehalle. “Mean Field Game of Controls and An Application To Trade Crowding”. In: Mathematics and Financial Economics (2019). url: https://hal.archives-ouvertes.fr/hal-01389128. René Carmona and François Delarue. “Probabilistic Analysis of Mean-Field Games”. In: SIAM Journal on Control and Optimization 51 (Oct. 2012). doi: 10.1137/120883499. René Carmona and François Delarue. Probabilistic Theory of Mean Field Games with Applications I-II. Springer, 2018. René Carmona and Mathieu Laurière. “Convergence Analysis of Machine Learning Algorithms for the Numerical Solution of Mean Field Control and Games: II – The Finite Horizon Case”. In: arXiv preprint arXiv:1908.01613 (Aug. 2019). Jean-Pierre Fouque and Zhaoyu Zhang. “Deep Learning Methods for Mean Field Control Problems with Delay”. In: arXiv preprint arXiv:1905.00358 (May 2019). Maximilien Germain, Joseph Mikael, and Xavier Warin. “Numerical resolution of McKean-Vlasov FBSDEs using neural networks”. In: arXiv preprint arXiv:1908.00412 (2019). Jiequn Han, Arnulf Jentzen, and Weinan E. “Solving high-dimensional partial differential equations using deep learning”. In: Proceedings of the National Academy of Sciences 115 (July 2017). doi: 10.1073/pnas.1718942115. Minyi Huang, Roland Malhame, and Peter Caines. “Large population stochastic dynamic games: Closed-loop McKean-Vlasov systems and the Nash certainty equivalence principle”. In: Commun. Inf.

  • Syst. 6 (Jan. 2006). doi: 10.4310/CIS.2006.v6.n3.a5.

Côme Huré, Huyên Pham, and Xavier Warin. “Some machine learning schemes for high-dimensional nonlinear PDEs”. In: Math. Comp. (2020). To appear. Jean-Michel Lasry and Pierre-Louis Lions. “Jeux à champ moyen. II – Horizon fini et contrôle

  • ptimal”. In: Comptes Rendus. Mathématique. Académie des Sciences, Paris 10 (Nov. 2006). doi:

10.1016/j.crma.2006.09.018. Anis Matoussi, Clémence Alasseur, and Imen Ben Taher. “An Extended Mean Field Game for Storage in Smart Grids”. 27 pages, 5 figures. Mar. 2018. url: https://hal.archives-ouvertes.fr/hal-01740707. Justin Sirignano and Konstantinos Spiliopoulos. “DGM: A deep learning algorithm for solving partial differential equations”. In: J. Computational Phys. 375 (Aug. 2017). doi: 10.1016/j.jcp.2018.08.029. Maximilien Germain ML for McKean-Vlasov FBSDEs 8th January 2020 18 / 18