Mean field limit of controlled system: From discrete to continuous - - PowerPoint PPT Presentation

Mean field limit of controlled system:

From discrete to continuous problems. Nicolas Gast1

EPFL LIG and INRIA

January 25, 2011

1Joint work with Bruno Gaujal (INRIA) and Jean-Yves Le Boudec (EPFL)

Introduction: mean field interacting objects

The term “mean field” applies for system of interacting objects: communication networks, computing clusters. epidemic models, gossip. chemical reactions.

Introduction Stochastic approximation Optimal mean field Non-smooth dynamics Conclusion 2/25

Introduction: mean field interacting objects

The term “mean field” applies for system of interacting objects: communication networks, computing clusters. epidemic models, gossip. chemical reactions. Objectives Analyze and improve the performance of the system: Characterize the dynamics of the system Find good (or optimal) policies to control the system. Tools Build a stochastic (microscopic) model of interacting objects. Study macroscopic properties. Fight curse of dimensionality via continuous approximation.

Introduction Stochastic approximation Optimal mean field Non-smooth dynamics Conclusion 2/25

Example of mean field approximation: SIR model

a Susceptible individual becomes infected with rate γI an Infected recovers with rate β a Recovered becomes susceptible with rate µ. Two main characteristics

Introduction Stochastic approximation Optimal mean field Non-smooth dynamics Conclusion 3/25

Example of mean field approximation: SIR model

a Susceptible individual becomes infected with rate γI an Infected recovers with rate β a Recovered becomes susceptible with rate µ. Two main characteristics 1 Exchangeable objects – The important quantity is the number of

• bjects in each state MN(t)

0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.1 0.2 0.3 0.4 0.5 0.6 0.7 N=100

Introduction Stochastic approximation Optimal mean field Non-smooth dynamics Conclusion 3/25

Example of mean field approximation: SIR model

a Susceptible individual becomes infected with rate γI an Infected recovers with rate β a Recovered becomes susceptible with rate µ. Two main characteristics 1 Exchangeable objects – The important quantity is the number of

• bjects in each state MN(t)

2 The drift – average difference of MN(t) between t and t + dt.

0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.1 0.2 0.3 0.4 0.5 0.6 0.7 N=100 drift

The drift is: f (m) =   −γIS + µR γIS − βI βI − µR  

Introduction Stochastic approximation Optimal mean field Non-smooth dynamics Conclusion 3/25

Example of mean field approximation: SIR model

a Susceptible individual becomes infected with rate γI an Infected recovers with rate β a Recovered becomes susceptible with rate µ. Two main characteristics 1 Exchangeable objects – The important quantity is the number of

• bjects in each state MN(t)

2 The drift – average difference of MN(t) between t and t + dt.

0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.1 0.2 0.3 0.4 0.5 0.6 0.7 N=100 drift mean field approximation

The drift is: f (m) =   −γIS + µR γIS − βI βI − µR   The mean field approximation is the solution of ˙ m = f (m). Question: what is the link between MN(t) and m(t)?

Introduction Stochastic approximation Optimal mean field Non-smooth dynamics Conclusion 3/25

Mean field for performance evaluation.

Generic tool to study multiple properties: Transient behavior

Starting from MN(0) = m(0), is MN(t) close to m(t) for all t? Yes if f has some continuity properties ((Le Boudec, Bena¨ ım 08)).

Steady state behavior

If limt→∞ m(t) = m∗, is the stationary distrib. of MN(t) close to m∗? “Yes, but”. Ex: load balancing (Mitzenmacher 98, Gast et. al. 10).

Propagation du chaos (Sznitman 91, Graham 00):

When the size of the system grows, objects become independent. Ex: 802.11 (Bianchi 00, Bordenave et al. 05)

Stability

Stability of the fluid implies stability of the stochastic system. True for some models (Ex: Dai 95, Bramson 08)

Introduction Stochastic approximation Optimal mean field Non-smooth dynamics Conclusion 4/25

Mean field for performance evaluation.

Generic tool to study multiple properties: Transient behavior

Starting from MN(0) = m(0), is MN(t) close to m(t) for all t? Yes if f has some continuity properties ((Le Boudec, Bena¨ ım 08)).

Steady state behavior

If limt→∞ m(t) = m∗, is the stationary distrib. of MN(t) close to m∗? “Yes, but”. Ex: load balancing (Mitzenmacher 98, Gast et. al. 10).

Propagation du chaos (Sznitman 91, Graham 00):

When the size of the system grows, objects become independent. Ex: 802.11 (Bianchi 00, Bordenave et al. 05)

Stability

Stability of the fluid implies stability of the stochastic system. True for some models (Ex: Dai 95, Bramson 08)

In this talk: extensions to controlled system. Optimal control problems. Non-smooth dynamics.

Introduction Stochastic approximation Optimal mean field Non-smooth dynamics Conclusion 4/25

Outline

• 1. Introduction
• 2. Mean field via stochastic approximation

◮ Basic hypothesis and definition.

• 3. Optimal control of mean field models

◮ How to define the limiting deterministic optimization problem? ◮ Convergence results. ◮ Applicability.

• 4. Going further: non-smooth dynamics

◮ if the drift f is not continuous: can we define an ODE dm

dt = f (m)?

◮ What is the limit of MN?

• 5. Conclusion and references

Introduction Stochastic approximation Optimal mean field Non-smooth dynamics Conclusion 5/25

Outline

• 1. Introduction
• 2. Mean field via stochastic approximation

◮ Basic hypothesis and definition.

• 3. Optimal control of mean field models

◮ How to define the limiting deterministic optimization problem? ◮ Convergence results. ◮ Applicability.

• 4. Going further: non-smooth dynamics

◮ if the drift f is not continuous: can we define an ODE dm

dt = f (m)?

◮ What is the limit of MN?

• 5. Conclusion and references

Introduction Stochastic approximation Optimal mean field Non-smooth dynamics Conclusion 5/25

Mean field convergence via stochastic approximation

Assumptions [Bena¨ ım, Le Boudec 08] A1 Objects are exchangeable

(i.e. MN(t) is Markovian.)

A2 Number of objects doing a transition at a given time step is less than NIN (in average) andN2I 2

N (in second moment).

Typically, there are at most K objects doing a transition. In that case, IN = K/N works.

A3 The drift f is lipschitz-continuous. f (m) = 1 N E

• MN(t + 1

N ) − MN(t) | MN(t) = m

• .

(classical assumption but annoying.)

Introduction Stochastic approximation Optimal mean field Non-smooth dynamics Conclusion 6/25

Mean field convergence via stochastic approximation

A1 Objects are exchangeable A2 Number of objects doing a transition at a given time step is less than NIN (in average) andN2I 2

N (in second moment).

A3 The drift f is lipschitz-continuous.

Theorem (Bena¨ ım-Le Boudec 08) Under assumptions (A1,A2,A3), sup

0≤t≤T

• MN(t) − m(t)
• P

− → 0. where m is the solution of the ODE ˙ m = f (m). Theorem (BB 08) If the ODE ˙ m = f (m) has a unique attractor m∗. The stationary distribution πN of MN converges to δm∗. If MN is picked according to πN, then any k objects are independent as N → ∞.

Introduction Stochastic approximation Optimal mean field Non-smooth dynamics Conclusion 6/25

Idea of the proof:

By definition of the drift, MN

π (t + 1 N ) can be written:

MN(t + 1 N ) = MN(t) + 1 N

• f (MN(t))
• Drift (deterministic)

+ noise

Random, E[.]=0

• .

At time t = k 1

N , MN(t) is equal to:

MN(t) = MN

0 + k−1

• i=0

1 N f (MN( i N ))

• Euler discretization

+ 1 N

k

• i=0

noise

• Converges to 0

. Convergence of the noise: martingale argument. Convergence of the Euler discretization: by Gronwall’s lemma.

Introduction Stochastic approximation Optimal mean field Non-smooth dynamics Conclusion 7/25

Application example: SIR model

a Susceptible individual becomes infected with rate γI an Infected recovers with rate β a Recovered becomes susceptible with rate µ. This is a mean field model Symmetry: Objects are indistinguishable. Intensity: each transition affects one (recovery) or two objects (infection). Drift continuous: OK. The drift is:   −γIS + µR γIS − βI βI − µR  

Introduction Stochastic approximation Optimal mean field Non-smooth dynamics Conclusion 8/25

Application example: SIR model

a Susceptible individual becomes infected with rate γI an Infected recovers with rate β a Recovered becomes susceptible with rate µ. This is a mean field model Symmetry: Objects are indistinguishable. Intensity: each transition affects one (recovery) or two objects (infection). Drift continuous: OK. The drift is:   −γIS + µR γIS − βI βI − µR  

0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.1 0.2 0.3 0.4 0.5 0.6 0.7 N=100 drift mean field approximation 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.1 0.2 0.3 0.4 0.5 0.6 0.7 N=100 drift mean field approximation Introduction Stochastic approximation Optimal mean field Non-smooth dynamics Conclusion 8/25

Outline

• 1. Introduction
• 2. Mean field via stochastic approximation

◮ Basic hypothesis and definition.

• 3. Optimal control of mean field models

◮ How to define the limiting deterministic optimization problem? ◮ Convergence results. ◮ Applicability.

• 4. Going further: non-smooth dynamics

◮ if the drift f is not continuous: can we define an ODE dm

dt = f (m)?

◮ What is the limit of MN?

• 5. Conclusion and references

Introduction Stochastic approximation Optimal mean field Non-smooth dynamics Conclusion 9/25

High dimension stochastic optimization problem

At time t, a controller chooses an action at ∈ A. Goal of the controller: find a policy π : X → A that minimizes a cost function. Example: Take the SIR model People can vaccinated or treated. Nothing is free:

Vaccination or treatment has a cost. An infected person has a cost.

Problem: Many resources and applications. → State space of the system is huge.

Introduction Stochastic approximation Optimal mean field Non-smooth dynamics Conclusion 10/25

A markov decision problem

The theoretical framework is well-known (MDP). Contrˆ

• leur central

a0 a1 aT−1 [X(0)] [X(1)] [X(T)]

rand(a0) rand(a1) rand(aT−1)

State of the system at time t is X(t)

1

A centralized controller modifies the dynamics of the system.

Introduction Stochastic approximation Optimal mean field Non-smooth dynamics Conclusion 11/25

A markov decision problem

The theoretical framework is well-known (MDP). Contrˆ

• leur central

a0 a1 aT−1 [X(0)] [X(1)] [X(T)]

rand(a0) rand(a1) rand(aT−1)

Cost(X) Cost(X) + . . . + State of the system at time t is X(t)

1

A centralized controller modifies the dynamics of the system.

2

At state t is associated a cost cost(X(t)).

Introduction Stochastic approximation Optimal mean field Non-smooth dynamics Conclusion 11/25

A markov decision problem

The theoretical framework is well-known (MDP). Contrˆ

• leur central

a0 a1 aT−1 [X(0)] [X(1)] [X(T)]

rand(a0) rand(a1) rand(aT−1)

Cost(X) Cost(X) + . . . + State of the system at time t is X(t)

1

A centralized controller modifies the dynamics of the system.

2

At state t is associated a cost cost(X(t)). Goal of the controller: find the best policy π∗ : X → A to minimize the average cost (over a finite horizon of time).

Introduction Stochastic approximation Optimal mean field Non-smooth dynamics Conclusion 11/25

A markov decision problem

The theoretical framework is well-known (MDP). Contrˆ

• leur central

a0 a1 aT−1 [X(0)] [X(1)] [X(T)]

rand(a0) rand(a1) rand(aT−1)

Cost(X) Cost(X) + . . . + State of the system at time t is X(t)

1

A centralized controller modifies the dynamics of the system.

2

At state t is associated a cost cost(X(t)). Goal of the controller: find the best policy π∗ : X → A to minimize the average cost (over a finite horizon of time). E T

• t=1

cost(Xπ(t))

• Introduction

Stochastic approximation Optimal mean field Non-smooth dynamics Conclusion 11/25

A markov decision problem

The theoretical framework is well-known (MDP). Contrˆ

• leur central

a0 a1 aT−1 [X(0)] [X(1)] [X(T)]

rand(a0) rand(a1) rand(aT−1)

Cost(X) Cost(X) + . . . + State of the system at time t is X(t)

1

A centralized controller modifies the dynamics of the system.

2

At state t is associated a cost cost(X(t)). Goal of the controller: find the best policy π∗ : X → A to minimize the average cost (over a finite horizon of time). V N

∗ (X(0)) = inf π

E T

• t=1

cost(Xπ(t))

• Introduction

Stochastic approximation Optimal mean field Non-smooth dynamics Conclusion 11/25

Mean field controlled system

A mean field controlled system is described by: Symmetric system with N objects. Intensity IN and drift f (a, ·)

+

A controller: chooses an action a ∈ A. Cost function: cost(m). If MN(t) denotes the proportion of objects in each state when applying a sequence of action A. We know that: MN

A (t) →N→∞ mA(t)

where m is a deterministic system (ODE).

Introduction Stochastic approximation Optimal mean field Non-smooth dynamics Conclusion 12/25

The limiting deterministic optimal control

If the sequence of action choosen is a(t), we have: State ma(t)

dma dt = f (ma(t), a(t))

ma(0) = m0. Cost va(m0) T

0 cost(ma(t))dt.

• ptimal cost

v∗(m0) = inf

{a|a piecewize lipschitz} va(m0).

Question: what is the relation between the stochastic optimal control V N

∗ and the deterministic optimal control v∗?

Convergence of value functions? Convergence of optimal policies? If a∗ be the optimal policy of the limit: va∗ = v∗: will a∗ will perform well on the stochastic system?

Introduction Stochastic approximation Optimal mean field Non-smooth dynamics Conclusion 13/25

Convergence results

Assumptions A1 Objects are exchangeable

(i.e. MN(t) is Markovian.)

A2 Number of objects doing a transition at a given time step is less than NIN (in average) andN2I 2

N (in second moment).

Typically, there are at most K objects doing a transition. In that case, IN = K/N works.

A3 All parameters (in particular the drift) are lipschitz-continuous in m and a

(classical assumption but annoying.)

Introduction Stochastic approximation Optimal mean field Non-smooth dynamics Conclusion 14/25

Convergence results

A1 Objects are exchangeable A2 Number of objects doing a transition at a given time step is less than NIN (in average) andN2I 2

N (in second moment).

A3 All parameters (in particular the drift) are lipschitz-continuous in m and a

Theorem ([GGL10]) For the finite horizon (or discounted) reward, if A1,A2,A3, + The optimal cost for the stochastic system converges to the

• ptimal cost of its deterministic limit:

V N

∗ N→∞

− − − − →v∗. + The optimal policy for the deterministic system a∗ is asymptotically optimal:

• V N

∗ − V N a∗

• N→∞

− − − − →0. Convergence holds in probability with explicit bounds. Second order results (CLT-like) for the discrete case. However: − πN

∗ might not converge.

− Deterministic limit might be hard.

Introduction Stochastic approximation Optimal mean field Non-smooth dynamics Conclusion 14/25

Optimal infection problem

Example [Khouzani et al, 10]: a worm wants to infect a computer network. The state of each node can be: Susceptible, infective, recovered or dead. The worm wants to optimize a damage function before time T (Action = killing an infected node). D(T) + T f (I(t))dt.

Introduction Stochastic approximation Optimal mean field Non-smooth dynamics Conclusion 15/25

Optimal infection problem

Example [Khouzani et al, 10]: a worm wants to infect a computer network. The state of each node can be: Susceptible, infective, recovered or dead. The worm wants to optimize a damage function before time T (Action = killing an infected node). D(T) + T f (I(t))dt. The model satisfies the assumptions: 1 Objects exchangeable: OK. 2 Number of objects changing states: ≤ 2: OK. 3 Continuity of the parameters: OK.

Introduction Stochastic approximation Optimal mean field Non-smooth dynamics Conclusion 15/25

Optimal infection problem

Example [Khouzani et al, 10]: a worm wants to infect a computer network. The state of each node can be: Susceptible, infective, recovered or dead. The worm wants to optimize a damage function before time T (Action = killing an infected node). D(T) + T f (I(t))dt. The model satisfies the assumptions: 1 Objects exchangeable: OK. 2 Number of objects changing states: ≤ 2: OK. 3 Continuity of the parameters: OK. Using Pontryagin’s maximum principle, authors show that a bang-bang policy is optimal for the continuous case: Do not kill nodes before time t∗, Kill nodes after time t∗. Our results show that this is asymptotic optimal for the stochastic system.

Introduction Stochastic approximation Optimal mean field Non-smooth dynamics Conclusion 15/25

How to apply this in practice?

Stochastic system, N

• bjects

Mean field limit Optimal mean field Optimal stochastic system N → ∞ Deterministic Optimization Asymptotically

• ptimal

The complexity of the method depends on the complexity of the deterministic problem:

1

If we can solve the deterministic limit. (see paper for an example)

Deterministic optimal policy works well for the stochastic system.

2

Use properties of the deterministic optimal policies.

reduce the class of interesting policies.

3

“Simplifies” the brute-force approach

Problem simpler but still hard (HJB, dynamic programming)

Introduction Stochastic approximation Optimal mean field Non-smooth dynamics Conclusion 16/25

Outline

• 1. Introduction
• 2. Mean field via stochastic approximation

◮ Basic hypothesis and definition.

• 3. Optimal control of mean field models

◮ How to define the limiting deterministic optimization problem? ◮ Convergence results. ◮ Applicability.

• 4. Going further: non-smooth dynamics

◮ if the drift f is not continuous: can we define an ODE dm

dt = f (m)?

◮ What is the limit of MN?

• 5. Conclusion and references

Introduction Stochastic approximation Optimal mean field Non-smooth dynamics Conclusion 17/25

What happens when the drift is not continuous?

Boundary conditions (ex: queuing systems) When applying a policy to control the system (threshold effects). Example: Static priority Tasks of type 1 have priority. λ1 = 1 λ2 = 1 µ = 3

• Prop. tasks of type 1
• Prop. tasks of type 2

MN(t)

Introduction Stochastic approximation Optimal mean field Non-smooth dynamics Conclusion 18/25

What happens when the drift is not continuous?

Boundary conditions (ex: queuing systems) When applying a policy to control the system (threshold effects). Example: Static priority Tasks of type 1 have priority. λ1 = 1 λ2 = 1 µ = 3

• Prop. tasks of type 1
• Prop. tasks of type 2

m(t) ? MN(t)

Introduction Stochastic approximation Optimal mean field Non-smooth dynamics Conclusion 18/25

When the number of tasks of type 1 reaches 0: Drift not continuous: (−2, +1) if x > 0 (+1, −2) if x = 0 No trajectory m(t) satisfying dm

dt = f (m).

Idea: build a differential inclusion using convex closure.

Let F be the “convex closure” of f : F(m) def =

• ǫ>0

conv ({f (z) : z − m ≤ ǫ}) . The original ODE is replaced by: dm dt ∈ F(m) a.e.

x = 0

becomes

x = 0

Drift f Convex closure F

Introduction Stochastic approximation Optimal mean field Non-smooth dynamics Conclusion 19/25

A general convergence result

Assumptions A1 Objects are exchangeable

Same condition.

A2 Number of objects doing a transition at a given time step is less than NIN (in average) andN2I 2

N (in second moment).

Same condition.

A3 Continuity is replaced by A3bis f is bounded : f (x) ≤ C

Introduction Stochastic approximation Optimal mean field Non-smooth dynamics Conclusion 20/25

A general convergence result

A1 Objects are exchangeable A2 Number of objects doing a transition at a given time step is less than NIN (in average) andN2I 2

N (in second moment).

A3bis f is bounded

Let D(m0) def = be the set of solutions of the DI dm dt ∈ F(m) with m(0) = m0 F the convex closure of the drift f . Theorem (GG10n) Under A1,A2, A3bis. ∀T > 0, inf

m∈D(m0)

sup

0≤t≤T

• MN(t) − m(t)
• ∞

P

− → 0. + no regularity assumption. − solution is not unique.

+ numerical solvers exist.

Introduction Stochastic approximation Optimal mean field Non-smooth dynamics Conclusion 20/25

Application on the static priority example.

Clients 1 Clients 2 The drift is easy to compute. The ODE has no solution. Computation of convex closure. The DI has a unique solution.

Introduction Stochastic approximation Optimal mean field Non-smooth dynamics Conclusion 21/25

x = 0

Application on the static priority example.

Clients 1 Clients 2 The drift is easy to compute. The ODE has no solution. Computation of convex closure. The DI has a unique solution. Computation of the convex closure simplifies the approach. Allows one to some uncertainty on the behavior. Shows the existence of a fluid limit for any controlled system.

Introduction Stochastic approximation Optimal mean field Non-smooth dynamics Conclusion 21/25

x = 0

Outline

• 1. Introduction
• 2. Mean field via stochastic approximation

◮ Basic hypothesis and definition.

• 3. Optimal control of mean field models

◮ How to define the limiting deterministic optimization problem? ◮ Convergence results. ◮ Applicability.

• 4. Going further: non-smooth dynamics

◮ if the drift f is not continuous: can we define an ODE dm

dt = f (m)?

◮ What is the limit of MN?

• 5. Conclusion and references

Introduction Stochastic approximation Optimal mean field Non-smooth dynamics Conclusion 22/25

Conclusion and perspective

What I did talk about:

1

Mean field can be applied for optimization problems

2

Discontinuous dynamics. What I did not talk about (see e.g. [GG09, GG10]) Steady state behavior When the limiting process is in discrete time. Speed of convergence in the non-continuous case. My goal here Apply (part of) these results to wireless problems, e.g. admission control with micro base-stations. Study stability problem via differential inclusions. If you have any question/comment/idea/suggestion, let me know.

Introduction Stochastic approximation Optimal mean field Non-smooth dynamics Conclusion 23/25

References corresponding to this work:

Work stealing :

• N. Gast, B. Gaujal –

A Mean Field Model of Work Stealing in Large-Scale Systems – Sigmetrics 2010 – [GG10ws]

Optimization and mean field, discrete time limit:

• N. Gast, B. Gaujal –

A Mean Field Approach for Optimization in Particles Systems and Applications – ValueTools 2009 – [GG09]

• N. Gast, B. Gaujal – A Mean Field Approach for Optimization in Discrete

Time – DEDS 2010 – [GG10]

Optimization and mean field, continuous time limit:

• N. Gast, B. Gaujal, J.-Y. Le Boudec – Mean field for Markov Decision

Processes: from Discrete to Continuous Optimization – Inria RR 7239 –

[GGL10]

Non-smooth mean field:

• N. Gast, B. Gaujal – Mean field Limit of Non-Smooth Systems and

Differential Inclusions – MAMA 2010

• N. Gast, B. Gaujal –

Mean field limit of non-smooth systems: a differential inclusion limit. – Inria RR 7315 – [GG10n]

Introduction Stochastic approximation Optimal mean field Non-smooth dynamics Conclusion 24/25

Other references

Kurtz 70 – Solutions of ordinary differential equations as limits of pure jump Markov processes – Sznitman 89 – Topics in propagation of chaos – Mitzenmacher 98 – Analyses of load stealing models based in differential equations – Graham 00 – Chaoticity on path space for a queueing network with selection of the shortest queue among several – Bianchi 02 – Performance analysis of the IEEE 802 11 distributed coordination function – Bordenave, Mc Donald, Prouti` ere 05 – Random multi-access algorithms: A mean field analysis – Bena¨ ım-Le Boudec 08 – A class of mean field interaction models for computer and communication systems – Darling, Norris 08 – Differential equation approximations for Markov chains –

Introduction Stochastic approximation Optimal mean field Non-smooth dynamics Conclusion 25/25