[PPT] - On Ergodic Impulse Control with Constraint Maurice Robin Based on PowerPoint Presentation

SLIDE 1

On Ergodic Impulse Control with Constraint

Maurice Robin

Based on joint papers with J.L. Menaldi University Paris-Sanclay

91190 Saint-Aubin, France (e-mail: maurice.robin@polytechnique.edu)

IMA, Minneapolis, MN May 7–11, 2018

MR (UP-S) Ergodic Impulse Control with Constraint 1 / 19

SLIDE 2

Content

Statement of the problem
HJB equation
Solution of the HJB equation
Existence of an optimal control
Extensions
References

MR (UP-S) Ergodic Impulse Control with Constraint 2 / 19

SLIDE 3

Introduction Statement of the Problem

Statement of the Problem

(as in JL Menaldi’s talk except for the cost and ergodic assumptions)

The uncontrolled state is described by a Markov-Feller process xt (values

in E metric compact)

impulse control ν = (θi, ξi)i≥1, θi increasing sequence of stopping times,

ξi E valued random variable

constraint on impulse controls: θi > 0 and θi is a jump time of the signal

process yt yτn = 0, yt = t − τn for τn ≤ t ≤ τn+1, n ≥ 1, Tn = τn+1 − τn, conditionally to xt as IID random variables with intensity λ(x, y)

ξi ∈ Γ(xθi), Γ(x) closed set of E and ∀ξ ∈ Γ(x), Γ(ξ) ⊂ Γ(x)

MR (UP-S) Ergodic Impulse Control with Constraint 3 / 19

SLIDE 4

Introduction Statement of the Problem (2)

Statement of the Problem (2)

running cost f (x, y) and impulse cost c(x, ξ), both positive bounded and

continuous, c(x, ξ) ≥ c0 > 0 and c(x, ξ) + c(ξ, ξ′) ≥ c(x, ξ′)

Mg(x) ≡ infξ∈Γ(x){c(x, ξ) + g(x)} is assumed to be continuous if g is

continuous and there exists a measurable selector ˆ ξ(x, g). V will denote the set of admissible controls V = {(θi, ξi), i ≥ 1, θ1 > 0, yθi = 0} V0 the set of admissible controls satisfying the constraint, but θ1 = 0 is allowed

MR (UP-S) Ergodic Impulse Control with Constraint 4 / 19

SLIDE 5

Introduction Statement of the Problem (3)

Statement of the Problem (3)

Controlled process

The controlled process for a control ν is defined

n the product space Ω∞, Ω = D(R+; E × R+)

by a probability Pν

xy,

and (xν

t , yν t ) = (xi t, yi t) for θi−1 ≤ t < θi

evolves as the uncontrolled process between impulses instants.

MR (UP-S) Ergodic Impulse Control with Constraint 5 / 19

SLIDE 6

Introduction Statement of the Problem (4)

Statement of the Problem (4)

The average cost to be minimized:

J(x, y, ν) = lim inf

T→∞

1 T Eν

xy

T f (xν

s , yν s )ds +

i

✶θi≤Tc(xi−1

θi

, ξi)

µ(x, y) = inf{J(x, y, ν) : ν ∈ V}

We will use an auxiliary problem ˜ J(x, y, ν) = lim inf

n→∞

1 Eν

xyτn

Eν

xy

τn f (xν

s , yν s )ds +

i

✶θi≤Tc(xi−1

θi

, ξi)

µ0(x, y) = inf{J(x, y, ν) : ν ∈ V0}

MR (UP-S) Ergodic Impulse Control with Constraint 6 / 19

SLIDE 7

Introduction Additional Assumptions

Additional Assumptions λ(x, y) is ≥ 0 bounded and continuous and 0 < a1 ≤ Ex0(τ1) ≤ a2 Ergodicity assumption: P(x, B) = Ex0✶B(xτ1) ∀B ∈ B(E) satisfies: there exists a positive measure m on E s.t. 0 < m(E) ≤ 1 and P(x, B) ≥ m(B) ∀B ∈ B(E) Example: reflected diffusions and reflected diffusions with jumps for which the transition density satisfies p(x, t, x′) ≥ k(ε)

n

E × [ε, ∞[×E

MR (UP-S) Ergodic Impulse Control with Constraint 7 / 19

SLIDE 8

Solution HJB Equation

HJB Equation Heuristic argument with the discounted problem uα

0 (x, 0) = min

Muα

0 (x, 0), Ex0

τ1 e−αtf dt + e−ατ1uα

0 (xτ1, 0)

Let mα = inf uα

0 (x, 0), wα 0 = uα 0 − mα,

wα

0 (x, 0) = min

Mwα

0 (x, 0), Ex0

τ1 e−αt(f −αmα)dt+e−ατ1wα

0 (xτ1, 0)

Assuming wα

0 → w0 a function, and αmα → µ0 a constant,

w0(x, 0) = min

Mw0(x, 0), Ex0

τ1 (f − µ0)dt + w0(xτ1, 0)

One can also use heuristic argument on

uT

0 (t, x, y) = inf Eν x0

T−t f dt +

i

✶θi≤T−tc(xi−1

θi

, ξi)

.

MR (UP-S) Ergodic Impulse Control with Constraint 8 / 19

SLIDE 9

Solution HJB Equation

HJB Equation (2)

For the auxiliary problem:

Find (w0, µ0), µ0 constant, such that w0(x, 0) = min

Mw0(x, 0), Ex0

τ1 [f − µ0]ds + w0(xτ1, 0)

then w0(x, y), for y > 0, is given by

w0(x, y) = Exy τ1 [f − µ0]ds + w0(xτ1, 0)

For the initial problem: (w0, µ0) gives w(x, y) as

w(x, y) = Exy τ1 [f − µ0]ds + w0(xτ1, 0)

MR (UP-S)

Ergodic Impulse Control with Constraint 9 / 19

SLIDE 10

Solution Solution (µ0, w0)

Solution (µ0, w0)

A discrete time HJB equation for
µ0, w0(x, 0)
: define

ℓ(x) = Ex0 τ1 f (xs, ys)ds

,

Pg(x) = Ex0g(xτ1) τ(x) = Ex0τ1, w0(x) ≡ w0(x, 0), then w0(x) = min

inf

ξ∈Γ(x)

c(x, ξ) + w0(ξ)
, ℓ(x) − µ0τ(x) + Pw0(x)
is equivalent to the previous HJB equation

Proposition There exists a solution (µ0, w0) in R+ × C(E) of the HJB equation. Remark: If w0 is solution, w0 + constant is also solution. The uniqueness

f µ0 will come from the stochastic interpretation.

MR (UP-S) Ergodic Impulse Control with Constraint 10 / 19

SLIDE 11

Solution Solution (µ0, w0)

Arguments The proof uses the following equivalent form of the HJB equation w0(x) = inf

ξ∈Γ(x)∪{x}

ℓ(ξ) + ✶ξ=xc(x, ξ) − µ0τ(ξ) + Pw0(ξ)
= Rw0

and the fact that P(x, β) ≥ τ(x)γ(β) for a positive measure γ on E satisfying γ(E) > 1−β

τ(x), 0 < β < 1.

Then R is a contraction on C(E). w0 is the unique fixed point and µ0 =

E

w0(x)γ(dx)

MR (UP-S) Ergodic Impulse Control with Constraint 11 / 19

SLIDE 12

Solution Existence of an Optimal Control

Existence of an Optimal Control Additional assumptions: the (uncontrolled) Markov process (xt, yt) has a unique invariant measure ζ and there exists a continuous function h(x, y) s.t. Exyh(xτ, yτ) = h(x, y) − Exy τ [f (xt, yt) − ¯ f ]dt

,

for any finite stopping time τ, with ¯ f =

E×R+ f (x, y)dζ.

Remark: if f (x, y) = f (x), then it is sufficient to assume that the Poisson equation for xt, i.e. −Axh = f (x) − ¯ f has a continuous solution.

MR (UP-S) Ergodic Impulse Control with Constraint 12 / 19

SLIDE 13

Solution Existence of an Optimal Control (2)

Existence of an Optimal Control (2)

Theorem With the additional assumption, we have µ0 = inf ˜ J(x, 0, ν), ν ∈ V0

and there exists an optimal control ˆ

ν0 of the auxiliary problem case 1. µ0 = ¯ f : then it is optimal to “do nothing” case 2. µ0 < ¯ f : one can rewrite the HJB equation ˜ w(x) = min ˜ ψ(x), ˜ ℓ(x) + P ˜ w(x)

with ˜

w = w0 − h(x, 0), ˜ ψ = Mw0 − h(x, 0), ˜ ℓ(x) = (¯ f − µ0)Ex0τ1. This is the HJB equation of a discrete optimal stopping problem which has an optimal control ˆ η = inf

n ≥ 0 : ˜

w(xn) = ˜ ψ(xn)

, i.e., ˆ

η = inf

n ≥ 0 : w0(xn) = Mw0(xn)
where

xn is the Markov chain xτn. From this, we deduce an optimal control with θ1 = τˆ

η, and θi = θˆ ηi with ˆ

ηi = inf

n ≥ ˆ

ηi : w0(xn) = Mw0(xn)

.

MR (UP-S) Ergodic Impulse Control with Constraint 13 / 19

SLIDE 14

Solution Existence of an Optimal Control (3)

Existence of an Optimal Control (3) Corollary µ0 = inf ˜ J(x, y, ν), v ∈ V

and the optimal control ˆ

ν obtained by translation by τ1 of the control ˆ ν0. The final result is given by Theorem µ0 = inf{J(x, y, ν) : ν ∈ V} = J(x, y, ˆ ν)

MR (UP-S) Ergodic Impulse Control with Constraint 14 / 19

SLIDE 15

Solution Existence of an Optimal Control (4)

Existence of an Optimal Control (4) A first step is to prove: Proposition (µ0, w0) being the solution previously obtained and recalling that w(x, y) = Ex0 τ1 [f − µ0]ds + w0(xτ1, 0)

(µ0, w) is solution of

−Axyw(x, y) + λ(x, y)[w(x, 0) − Mw(x, 0)]+ = f − µ0. where Axy is the (weak) infinitesimal generator of the uncontrolled process Axyϕ = Axϕ + ∂ϕ ∂y + λ(x, y)

ϕ(x, 0) − ϕ(x, y)
.

MR (UP-S) Ergodic Impulse Control with Constraint 15 / 19

SLIDE 16

Solution Existence of an Optimal Control (5)

Existence of an Optimal Control (5) To prove the proposition, one first shows w0(x) = min{w(x, 0), Mw(x, 0)} which gives w(x, y) = Exy τ1 [f − µ0]dt + w(xτ1, 0) − [w(xτ1, 0) − Mw]+ from which we deduce that equation.✷ Next, the proposition allows us to show that MT = T [f (xt, yt) − µ0]ds + w(xT, yT) is a submartingale. This gives µ0 ≤ J(x, y, ν), ∀ ν ∈ V, and, from the first expression of w(x, y), on obtains µ0 = J(x, y, ν).

MR (UP-S) Ergodic Impulse Control with Constraint 16 / 19

SLIDE 17

Solution Existence of an Optimal Control (5)

Extensions E locally compact.

If we assume Φ(t)C0(E) ⊂ C0(E) all other assumptions being

unchanged, one can still obtain the results on the HJB equation.

However, the “additional assumption” is no longer sufficient to get the

result on the existence of an optimal cost.

adding “h(x, 0) bounded” would be sufficient (e.g., if −Axyh = f − ¯

f has a bounded solution), but more general assumptions would require further work.

When λ is independent of x and f independent of y, one can obtain an
ptimal control without an additional assumption.

MR (UP-S) Ergodic Impulse Control with Constraint 17 / 19

SLIDE 18

Solution Existence of an Optimal Control (5)

Extensions (2)

P(x, B) ≥ m(B) is also restrictive for E locally compact

Replacing by a “localized” condition like P(x, B) ≥ α✶K(x)m(B), allows to obtains the results on the HJB equations and, if h(x, 0) is bounded, the existence of an optimal control.

MR (UP-S) Ergodic Impulse Control with Constraint 18 / 19

SLIDE 19

Solution References (among others)

References (among others) Besides the classical works on impulse control,

A. Bensoussan and J.L. Lions (book 1982), etc, . . . ,
For discrete time control: A. Bensoussan (book, 2011),
O. Hernandez-Lerma and J. Lasserre (books 1996, 1999), . . .
For average control, ergodicity: Bensoussan (book 1988),
M. Kurano (paper 1989),
F. Luque-Vasquez and O. Hernandez-Lerma (paper 1999),
L. Stettner (paper 1986), D. Gatarek and L. Stettner (paper 1990),
A. Arapostathis et al. (survey paper 1993),

M.G. Garroni and J.L. Menaldi (book 2002), Several papers with J.L. Menaldi (1997, 2013, . . . ), Papers to appear with J.L. Menaldi, H. Jasso-Fuentes, T. Prieto-Rumeau.

MR (UP-S) Ergodic Impulse Control with Constraint 19 / 19