Optimal Control in the space of probability measures Claudia Totzeck - - PowerPoint PPT Presentation

CROWDS

Optimal Control in the space of probability measures Claudia Totzeck

joint work with

• M. Burger, R. Pinnau and O.Tse

Crowds - Models and Control 3./7. June 2019 CIRM Marseille, France

CROWDS

Deterministic Optimization Problem Stochastic Optimization Problem

Optimal Control Problem

We consider an OC problem subject to an evolution equation of Vlasov-type

General OCP

min

C([0,T],P2(Rd ))×Uad

J(µ, u) = J1(µ) + J2(u) s.t. ∂tµt + ∇x · (v(µt, ut)µt) = 0. We assume for the cost functional that J1(µ) cylindrical, i.e. J1(µ) = j(g1, µ, . . . , gL, µ) with j ∈ C1(RL), gℓ ∈ C1(Rd), ℓ = 1, . . . , L such that gℓ, µ =

• Rd gℓdµ < ∞ and |∇gℓ| ≤ Cg(1 + |x|)

for all x ∈ Rd, ℓ = 1, . . . , L for some Cg > 0, J2 ∈ C1, w.l.s. and coercive.

C.Totzeck OC for probability measures

• 7. June 2019

2/ 32

CROWDS

Deterministic Optimization Problem Stochastic Optimization Problem

Optimal Control Problem

We consider an OC problem subject to an evolution equation of Vlasov-type

General OCP

min

C([0,T],P2(Rd ))×Uad

J(µ, u) = J1(µ) + J2(u) s.t. ∂tµt + ∇x · (v(µt, ut)µt) = 0. We assume for the velocity field v that v : P2(Rd) × RdM → Liploc(Rd) such that for all (µ, u) v(µ, u)(x) − v(µ, u)(y), x − y ≤ Cl|x − y|2 x, y ∈ Rd for Cl > 0 independent of (µ, u), for any two (µ, u), (µ′, u′) exists Cv > 0 independent of (µ, u), (µ′, u′) such that v(µ, u) − v(µ′, u′)sup ≤ Cv(W2(µ, µ′) + u − u′2).

C.Totzeck OC for probability measures

• 7. June 2019

3/ 32

CROWDS

Deterministic Optimization Problem Stochastic Optimization Problem

Application to have in mind: drones herding sheep

Source: https://www.youtube.com/watch?v=D8mXL2JapWM

C.Totzeck OC for probability measures

• 7. June 2019

4/ 32

CROWDS

Deterministic Optimization Problem Stochastic Optimization Problem

Questions Can we control the drones to guide the sheep?

to a desired location with desired variance?

What happens for many sheep? N → ∞?

is there a limiting behaviour? can we get convergence of the controls? even a rate?

What is the appropriate mathematical setup?

adjoint calculus in which sense?

Can we add noise to the dynamic of the sheep?

C.Totzeck OC for probability measures

• 7. June 2019

5/ 32

CROWDS

Deterministic Optimization Problem Stochastic Optimization Problem

Modelling the dynamics

consider a first order dynamic the crowd is modelled with the help of a probability measure µ: (0, T) × R2 → R, µ ∈ C((0, T), P2(R2)) the agents by (um)m=1,...,M =: u ∈ H1((0, T), R2M)

Special structure of v

We assume v(µt, ut) = −K1 ∗ µt − M

ℓ=1 K2(x − uℓ).

Leading to the state equation ∂tµt = ∇x ·

• (K1 ∗ µt +

M

• ℓ=1

K2(x − uℓ)) µt

• .

C.Totzeck OC for probability measures

• 7. June 2019

6/ 32

CROWDS

Deterministic Optimization Problem Stochastic Optimization Problem

Using the empirical measure µN(t, x) = 1 N

N

• i=1

δ(x − xi(t)) we get the

System of ODEs

˙ xi = − 1 N

• j=i

G1(xi, xj) − 1 M

• m

G2(xi, um), x(0) = x0, u(0) = u0, m = 1, . . . , M. µN based on a solution of the ODE is a weak solution of the PDE for random (xi

0)i=1,...,N with law(µ0), we have µN 0 → µ0 as N → ∞.1

passing to the mean-field limit yields µN

t → µt

∀ t ∈ [0, T].2

1Varadarajan 2Golse, Dobrushin, Braun-Hepp, Neunzert

C.Totzeck OC for probability measures

• 7. June 2019

7/ 32

CROWDS

Deterministic Optimization Problem Stochastic Optimization Problem

First-order optimality conditions (FOOC) for N < ∞

Adjoint system for N < ∞

d dt ξi

t = −∇xv(µN t , ut)(xi t)ξi t − 1

N

N

• j=1

(∇K1)(xj

t − xi t)ξj t + ∂ij(xt, ut)

supplemented with terminal conditions ξi

T = 0 for i = 1, . . . , N.

Moreover, we get the

Optimality condition for N < ∞

T NduJ2(ut)[hu

t ] − Duv(xt, ut)[hu t ] · ξtdt = 0 for all hu ∈ C∞ c ((0, T), R2M),

where ξt satisfies the adjoint system. How does the mean-field counterpart look like?

C.Totzeck OC for probability measures

• 7. June 2019

8/ 32

CROWDS

Deterministic Optimization Problem Stochastic Optimization Problem

Passing to the limit N → ∞

We assume 1 N

N

• i=1

∂ij(xt, ut) = δµj(µN

t , ut)

and define the empirical momentum mN

t = 1

N

N

• i=1

ξi

t δxi

t.

This leads to the adjoint equation ∂tmN

t + ∇ · (v(µN t , ut) ⊗ mN t ) = Ψ(µN t , ut)[mN t ],

Ψ(µN

t , ut)[mN t ] = −(∇xv)(µN t , ut)mN t − µN t ∇K1 ∗ mN t + µN t δµj(µN t ).

C.Totzeck OC for probability measures

• 7. June 2019

9/ 32

CROWDS

Deterministic Optimization Problem Stochastic Optimization Problem

L2 versus W2

Starting from the weak formulation 0 = e(µ, q) = qT, µt − T ∂tqt + v(µt, ut) · ∇qt, µtdt and assuming enough regularity, we find the L2−adjoint ∂tqt + v(µt, ut) · ∇qt =

• ∇qt(y) · K1(y − ·)µt(dy) + dµtj(µ)

Question: How is this one related to the W2−adjoint? ∂tmN

t + ∇ · (v(µN t , ut) ⊗ mN t ) = Ψ(µN t , ut)[mN t ],

Ψ(µN

t , ut)[mN t ] = −(∇xv)(µN t , ut)mN t − µN t ∇K1 ∗ mN t + µN t δµj(µN t ).

C.Totzeck OC for probability measures

• 7. June 2019

10/ 32

CROWDS

Deterministic Optimization Problem Stochastic Optimization Problem

Relation of L2- and W2-adjoints

W2-adjoint

∂tmN

t + ∇ · (v(µN t , ut) ⊗ mN t ) = Ψ(µN t , ut)[mN t ],

Ψ(µN

t , ut)[mN t ] = −(∇xv)(µN t , ut)mN t − µN t ∇K1 ∗ mN t + µN t δµj(µN t ).

If mt = ξtµt then

∂tξt + (v(µt, ut) · ∇)ξt = Ψ(µt, ut)[ξtµt]

If K1 = ∇φ1 and ξt = ∇qt then

∂tqt + v(µt, ut) · ∇qt =

• ∇qt(y) · K1(y − ·)µt(dy) + dµtj(µ)

C.Totzeck OC for probability measures

• 7. June 2019

11/ 32

CROWDS

Deterministic Optimization Problem Stochastic Optimization Problem

Summary of the links between the models

Other references in this direction

Consistent mean field optimality conditions for interacting agent systems (2018) Herty, Ringhofer Mean-field optimal control as Gamma-limit of finite agent controls (2018) Fornasier, Lisini, Orrieri, Savar´ e Mean-field control hierarchy (2016) Albi, Choi, Fornasier, Kalise Mean-field Pontryagin maximum principle (2015) Bongini, Fornasier, Rossi, Solombrino The Pontryagin Maximum Principle in the Wasserstein Space (2019) Bonnet, Rossi Mean-Field Sparse Optimal Control (2014) Fornasier, Piccoli, Rossi ... C.Totzeck OC for probability measures

• 7. June 2019

12/ 32

CROWDS

Deterministic Optimization Problem Stochastic Optimization Problem

Proposition3: W(µ0, µN

0 ) → 0 with rate CN−1/2 in expectation.

Theorem - Convergence of Controls

Let µ0 and µN

0 with µN 0 ⇀ µ0 and π0 an optimal transference plan of µN 0 and

µ0. We denote the optimal controls corresponding to µN

0 and µ0 as uN and u.

Then there exists σ3 and a constant K such that the following estimate holds uN − uL2((0,T),R2M) ≤ W(µ0, µN

0 )eKT.

Idea: problem in flow formulation, i.e. µt = (Zt)#µ0 derivation of the adjoint flow AT−t using standard L2-calculus regularity assumptions on G, H to show a Dobrushin-type inequality4 first order necessary optimality conditions yield the estimate.

3Fournier & Guillin 4Braun & Hepp, Dobrushin, Golse, Neunzert,...

C.Totzeck OC for probability measures

• 7. June 2019

13/ 32

CROWDS

Deterministic Optimization Problem Stochastic Optimization Problem

Numerics are based on the second order problem

J(µ, u) = 1 T T σ1 4 (V(µt) − ¯ V)2 + σ2 2 E(µt) − ¯ E(t)2

R2 + σ3

2M u2

R2M dt

with the help of the center of mass E and variance V of the crowd E(µt) =

• R4 x dµt(x, v),

V(µt) =

• R4(x − E(µt))2 dµt(x, v).

Optimisation problem

Find (µ∗, u∗) such that (µ∗, u∗) = argmin

(µ,u)

J(µ, u) subject to ∂tµ = −v · ∇xµ + ∇v ·

• G1 ∗ ρ + 1

M

M

• m=1

G2(x − dm) + αv

• µ
• ˙

dm = um, dm(0) = dm0, µt|t=0 = µ0, m = 1, . . . , M.

C.Totzeck OC for probability measures

• 7. June 2019

14/ 32

CROWDS

Deterministic Optimization Problem Stochastic Optimization Problem

Movie - Mean-Field Solution using Instantaneous Control

T = 200, M = 5, Ω = [−100, 100]2 × [−5, 5]2, hx = 8, hv = 2.5

C.Totzeck OC for probability measures

• 7. June 2019

15/ 32

CROWDS

Deterministic Optimization Problem Stochastic Optimization Problem

Numerical Results - The setting

t ∈ [0, 10], M = 5, N = 8000, Ω = [−100, 100]2 × [−5, 5]2 law(xi(0), vi(0)) = µ0 ∀i = 1, . . . , N.

C.Totzeck OC for probability measures

• 7. June 2019

16/ 32

CROWDS

Deterministic Optimization Problem Stochastic Optimization Problem

Numerics - Influence of the cost functional parameters

Figure: Focus on E(µt) − ¯ E2

R2

Figure: Focus on (V(µt) − ¯ V)2

These results are based on an instantaneous control approach.

C.Totzeck OC for probability measures

• 7. June 2019

17/ 32

CROWDS

Deterministic Optimization Problem Stochastic Optimization Problem

Numerics - Instantaneous Control vs. Optimal Control

Figure: Instantaneous Control Figure: Optimal Control

C.Totzeck OC for probability measures

• 7. June 2019

18/ 32

CROWDS

Deterministic Optimization Problem Stochastic Optimization Problem

Non-uniqueness of minima

σ1 = 0.005, σ2 = 0.5 σ1 = 0.005, σ2 = 0.5

different initial controls may lead to different optimal controls formation of the crowd resembles

C.Totzeck OC for probability measures

• 7. June 2019

19/ 32

CROWDS

Deterministic Optimization Problem Stochastic Optimization Problem

Controlling stochastic states

C.Totzeck OC for probability measures

• 7. June 2019

20/ 32

CROWDS

Deterministic Optimization Problem Stochastic Optimization Problem

Stochastic Model and Optimization Problem

Introducing some noise we obtain the following

System of SDEs

dxi = vidt, dvi =  − 1 N

• j=i

G1(xi, xj) − 1 M

• m

G2(xi, dm) − αvi   dt + σdBi

t,

˙ dm = um, x(0) = x0, v(0) = v0, d(0) = d0, m = 1, . . . , M. The corresponding PDE is given by ∂tµ = −v · ∇xµ + ∇v ·

• G1 ∗ ρ + 1

M

M

• m=1

G2(x − dm) + αv

• µ
• + σ2

2 ∆vµ, ˙ dm = um, dm(0) = dm0, µt|t=0 = µ0, m = 1, . . . , M.

C.Totzeck OC for probability measures

• 7. June 2019

21/ 32

CROWDS

Deterministic Optimization Problem Stochastic Optimization Problem

Cost functional

As cost functional we consider the variance around the desired destination: J(Y , u; ¯ Z, ¯ u) := T 1 2N

N

• k=1

E[Xk(t)] − ¯ Z(t)2 + γ 2 u(t) − ¯ u(t)2

RMD dt,

where E is the center of mass as above.

Additional parameters

In the space mapping framework the additional parameters ¯ Z and ¯ u are needed.

C.Totzeck OC for probability measures

• 7. June 2019

22/ 32

CROWDS

Deterministic Optimization Problem Stochastic Optimization Problem

Observations

stochastic forward system → how to compute adjoints? there are analytical results leading to a fully coupled forward-backward SDE system involving additional to the adjoint SDE a ghost process that captures the uncertain terminal condition5 numerically we have no clue how to compute it ( Monte-Carlo nightmare ?! ) ⇒ space mapping!

5Perkowski: Backward Stochastic Differential Equations: an Introduction

C.Totzeck OC for probability measures

• 7. June 2019

23/ 32

CROWDS

Deterministic Optimization Problem Stochastic Optimization Problem

Space mapping idea

stochastic particle model → fine model, control uf deterministic particle model → coarse model, control uc We assume that the coarse model is related to the fine model in such a way there exists an appropriate transformation T satisfying uc = T(uf ). The generation of this transformation T is the key element in the space mapping method. In fact, we want to approximate T.

C.Totzeck OC for probability measures

• 7. June 2019

24/ 32

CROWDS

Deterministic Optimization Problem Stochastic Optimization Problem

Space Mapping

Basic idea: find an approximate (coarse) model that is easy to optimize and fit it appropriately... Taken from: An Introduction to the Space Mapping Technique (2001) by Bakr, Bandler, Madsen, Søndergaard

C.Totzeck OC for probability measures

• 7. June 2019

25/ 32

CROWDS

Deterministic Optimization Problem Stochastic Optimization Problem

Approximation of space mapping function

Assuming the that the deterministic model is a reasonable approximation of the stochastic model, we obtain T(u∗

f ) = argmin uc∈Uad

1 2N

N

• i=1
• E[X i] − ¯

Z √γ(u − ¯ u)

• 2

L2(0,T)

≈ argmin

uc∈Uad

1 2N

N

• i=1
• xi − ¯

Z √γ(u − ¯ u)

• 2

L2(0,T)

= u∗

c .

Idea: Iteratively decrease the residual between the true deterministic

• ptimizer and the approximated stochastic optimizer.

C.Totzeck OC for probability measures

• 7. June 2019

26/ 32

CROWDS

Deterministic Optimization Problem Stochastic Optimization Problem

Aggressive Space Mapping Algorithm

set initial values and parameters initialize counter k = 0, approximate Jacobian B0 = I and tolerance ǫ, step size ρ Compute u0

f = u∗ c = argminuc J(Z, uc; Z, 0) subject to the deterministic

model While T(u∗

f ) − u∗ c > ǫSM

evaluate the expected center of mass ¯ X perform coarse model optimization T(uk

f ) = argmin uc

J(Y , uc; ¯ X) If k > 1 compute Bk = Bk−1 + ((T(uk

f ) − u∗ c ) ⊗ hk−1)/(hk−1)2

solve Bkhk = −(T(u∗

f ) − u∗ c ) for the update hk

update the control uk+1

f

= uk

f + ρhk

C.Totzeck OC for probability measures

• 7. June 2019

27/ 32

CROWDS

Deterministic Optimization Problem Stochastic Optimization Problem

Simulation

receding horizon control, i.e. optimize on subintervals desired destination is time dependent Z = Z(t) nonlinear conjugate gradient algorithm for optimization of coarse model

• nly coarse approximation of gradient in the inner loop

I1 u1

∗

[I1/2, I2/2] u2

∗

I2

Figure: Visualization of the receding horizon procedure.

C.Totzeck OC for probability measures

• 7. June 2019

28/ 32

CROWDS

Deterministic Optimization Problem Stochastic Optimization Problem

Simulation with 1-5 dogs, 100 sheep

Simulations with 1-5 dogs. Center of mass (blue), trajectories of the dogs (red), Stopping criterion based on the distance to Z = (−0.8, −0.8).

C.Totzeck OC for probability measures

• 7. June 2019

29/ 32

CROWDS

Deterministic Optimization Problem Stochastic Optimization Problem

Simulation with 3 and 4 dogs, 100 sheep

T = 200, Ik = 10k, N = 100, M = 3 or 4, dt = 0.0125, Z = (−0.8, −0.8)

C.Totzeck OC for probability measures

• 7. June 2019

30/ 32

CROWDS

Deterministic Optimization Problem Stochastic Optimization Problem

Summary

the optimal controls for the particle system converge to the optimal controls of the mean-field problem as N → ∞ the ’correct’ topology to derive the adjoints on the mean-field level is the Wasserstein-metric the corresponding adjoint is vector-valued and infeasible for numerical simulations we propose the L2-adjoint for numerical simulations to control the stochastic particle system we show that a space mapping approach works well

Outlook

for N ≫ 100 use mean-field limit as coarse model in space mapping ...

C.Totzeck OC for probability measures

• 7. June 2019

31/ 32

CROWDS

Deterministic Optimization Problem Stochastic Optimization Problem

Summary

the optimal controls for the particle system converge to the optimal controls of the mean-field problem as N → ∞ the ’correct’ topology to derive the adjoints on the mean-field level is the Wasserstein-metric the corresponding adjoint is vector-valued and infeasible for numerical simulations we propose the L2-adjoint for numerical simulations to control the stochastic particle system we show that a space mapping approach works well

Outlook

for N ≫ 100 use mean-field limit as coarse model in space mapping ...

C.Totzeck OC for probability measures

• 7. June 2019

31/ 32

CROWDS

Deterministic Optimization Problem Stochastic Optimization Problem

Baaaaaaaaaa!

Preprints on arXiv:

Burger, Pinnau, T., Tse

• Instantaneous control of interacting particle systems in the mean-field limit
• Mean-field optimal control and optimality conditions in the space of probability measures
• Controlling stochastic particle systems using space mapping (to appear in Springer Special Issue devoted to ECMI 2018)

totzeck@mathematik.uni-kl.de

C.Totzeck OC for probability measures

• 7. June 2019

32/ 32