On Ergodic Impulse Control with Constraint Maurice Robin Based on - - PowerPoint PPT Presentation

on ergodic impulse control with constraint
SMART_READER_LITE
LIVE PREVIEW

On Ergodic Impulse Control with Constraint Maurice Robin Based on - - PowerPoint PPT Presentation

On Ergodic Impulse Control with Constraint Maurice Robin Based on joint papers with J.L. Menaldi University Paris-Sanclay 91190 Saint-Aubin, France (e-mail: maurice.robin@polytechnique.edu) IMA, Minneapolis, MN May 711, 2018 MR (UP-S)


slide-1
SLIDE 1

On Ergodic Impulse Control with Constraint

Maurice Robin

Based on joint papers with J.L. Menaldi University Paris-Sanclay

91190 Saint-Aubin, France (e-mail: maurice.robin@polytechnique.edu)

IMA, Minneapolis, MN May 7–11, 2018

MR (UP-S) Ergodic Impulse Control with Constraint 1 / 19

slide-2
SLIDE 2

Content

  • Statement of the problem
  • HJB equation
  • Solution of the HJB equation
  • Existence of an optimal control
  • Extensions
  • References

MR (UP-S) Ergodic Impulse Control with Constraint 2 / 19

slide-3
SLIDE 3

Introduction Statement of the Problem

Statement of the Problem

(as in JL Menaldi’s talk except for the cost and ergodic assumptions)

  • The uncontrolled state is described by a Markov-Feller process xt (values

in E metric compact)

  • impulse control ν = (θi, ξi)i≥1, θi increasing sequence of stopping times,

ξi E valued random variable

  • constraint on impulse controls: θi > 0 and θi is a jump time of the signal

process yt yτn = 0, yt = t − τn for τn ≤ t ≤ τn+1, n ≥ 1, Tn = τn+1 − τn, conditionally to xt as IID random variables with intensity λ(x, y)

  • ξi ∈ Γ(xθi), Γ(x) closed set of E and ∀ξ ∈ Γ(x), Γ(ξ) ⊂ Γ(x)

MR (UP-S) Ergodic Impulse Control with Constraint 3 / 19

slide-4
SLIDE 4

Introduction Statement of the Problem (2)

Statement of the Problem (2)

  • running cost f (x, y) and impulse cost c(x, ξ), both positive bounded and

continuous, c(x, ξ) ≥ c0 > 0 and c(x, ξ) + c(ξ, ξ′) ≥ c(x, ξ′)

  • Mg(x) ≡ infξ∈Γ(x){c(x, ξ) + g(x)} is assumed to be continuous if g is

continuous and there exists a measurable selector ˆ ξ(x, g). V will denote the set of admissible controls V = {(θi, ξi), i ≥ 1, θ1 > 0, yθi = 0} V0 the set of admissible controls satisfying the constraint, but θ1 = 0 is allowed

MR (UP-S) Ergodic Impulse Control with Constraint 4 / 19

slide-5
SLIDE 5

Introduction Statement of the Problem (3)

Statement of the Problem (3)

  • Controlled process

The controlled process for a control ν is defined

  • n the product space Ω∞, Ω = D(R+; E × R+)

by a probability Pν

xy,

and (xν

t , yν t ) = (xi t, yi t) for θi−1 ≤ t < θi

evolves as the uncontrolled process between impulses instants.

MR (UP-S) Ergodic Impulse Control with Constraint 5 / 19

slide-6
SLIDE 6

Introduction Statement of the Problem (4)

Statement of the Problem (4)

  • The average cost to be minimized:

J(x, y, ν) = lim inf

T→∞

1 T Eν

xy

T f (xν

s , yν s )ds +

  • i

✶θi≤Tc(xi−1

θi

, ξi)

  • µ(x, y) = inf{J(x, y, ν) : ν ∈ V}

We will use an auxiliary problem ˜ J(x, y, ν) = lim inf

n→∞

1 Eν

xyτn

xy

τn f (xν

s , yν s )ds +

  • i

✶θi≤Tc(xi−1

θi

, ξi)

  • µ0(x, y) = inf{J(x, y, ν) : ν ∈ V0}

MR (UP-S) Ergodic Impulse Control with Constraint 6 / 19

slide-7
SLIDE 7

Introduction Additional Assumptions

Additional Assumptions λ(x, y) is ≥ 0 bounded and continuous and 0 < a1 ≤ Ex0(τ1) ≤ a2 Ergodicity assumption: P(x, B) = Ex0✶B(xτ1) ∀B ∈ B(E) satisfies: there exists a positive measure m on E s.t. 0 < m(E) ≤ 1 and P(x, B) ≥ m(B) ∀B ∈ B(E) Example: reflected diffusions and reflected diffusions with jumps for which the transition density satisfies p(x, t, x′) ≥ k(ε)

  • n

E × [ε, ∞[×E

MR (UP-S) Ergodic Impulse Control with Constraint 7 / 19

slide-8
SLIDE 8

Solution HJB Equation

HJB Equation Heuristic argument with the discounted problem uα

0 (x, 0) = min

  • Muα

0 (x, 0), Ex0

τ1 e−αtf dt + e−ατ1uα

0 (xτ1, 0)

  • Let mα = inf uα

0 (x, 0), wα 0 = uα 0 − mα,

0 (x, 0) = min

  • Mwα

0 (x, 0), Ex0

τ1 e−αt(f −αmα)dt+e−ατ1wα

0 (xτ1, 0)

  • Assuming wα

0 → w0 a function, and αmα → µ0 a constant,

w0(x, 0) = min

  • Mw0(x, 0), Ex0

τ1 (f − µ0)dt + w0(xτ1, 0)

  • One can also use heuristic argument on

uT

0 (t, x, y) = inf Eν x0

T−t f dt +

  • i

✶θi≤T−tc(xi−1

θi

, ξi)

  • .

MR (UP-S) Ergodic Impulse Control with Constraint 8 / 19

slide-9
SLIDE 9

Solution HJB Equation

HJB Equation (2)

  • For the auxiliary problem:

Find (w0, µ0), µ0 constant, such that w0(x, 0) = min

  • Mw0(x, 0), Ex0

τ1 [f − µ0]ds + w0(xτ1, 0)

  • then w0(x, y), for y > 0, is given by

w0(x, y) = Exy τ1 [f − µ0]ds + w0(xτ1, 0)

  • For the initial problem: (w0, µ0) gives w(x, y) as

w(x, y) = Exy τ1 [f − µ0]ds + w0(xτ1, 0)

  • MR (UP-S)

Ergodic Impulse Control with Constraint 9 / 19

slide-10
SLIDE 10

Solution Solution (µ0, w0)

Solution (µ0, w0)

  • A discrete time HJB equation for
  • µ0, w0(x, 0)
  • : define

ℓ(x) = Ex0 τ1 f (xs, ys)ds

  • ,

Pg(x) = Ex0g(xτ1) τ(x) = Ex0τ1, w0(x) ≡ w0(x, 0), then w0(x) = min

  • inf

ξ∈Γ(x)

  • c(x, ξ) + w0(ξ)
  • , ℓ(x) − µ0τ(x) + Pw0(x)
  • is equivalent to the previous HJB equation

Proposition There exists a solution (µ0, w0) in R+ × C(E) of the HJB equation. Remark: If w0 is solution, w0 + constant is also solution. The uniqueness

  • f µ0 will come from the stochastic interpretation.

MR (UP-S) Ergodic Impulse Control with Constraint 10 / 19

slide-11
SLIDE 11

Solution Solution (µ0, w0)

Arguments The proof uses the following equivalent form of the HJB equation w0(x) = inf

ξ∈Γ(x)∪{x}

  • ℓ(ξ) + ✶ξ=xc(x, ξ) − µ0τ(ξ) + Pw0(ξ)
  • = Rw0

and the fact that P(x, β) ≥ τ(x)γ(β) for a positive measure γ on E satisfying γ(E) > 1−β

τ(x), 0 < β < 1.

Then R is a contraction on C(E). w0 is the unique fixed point and µ0 =

  • E

w0(x)γ(dx)

MR (UP-S) Ergodic Impulse Control with Constraint 11 / 19

slide-12
SLIDE 12

Solution Existence of an Optimal Control

Existence of an Optimal Control Additional assumptions: the (uncontrolled) Markov process (xt, yt) has a unique invariant measure ζ and there exists a continuous function h(x, y) s.t. Exyh(xτ, yτ) = h(x, y) − Exy τ [f (xt, yt) − ¯ f ]dt

  • ,

for any finite stopping time τ, with ¯ f =

  • E×R+ f (x, y)dζ.

Remark: if f (x, y) = f (x), then it is sufficient to assume that the Poisson equation for xt, i.e. −Axh = f (x) − ¯ f has a continuous solution.

MR (UP-S) Ergodic Impulse Control with Constraint 12 / 19

slide-13
SLIDE 13

Solution Existence of an Optimal Control (2)

Existence of an Optimal Control (2)

Theorem With the additional assumption, we have µ0 = inf ˜ J(x, 0, ν), ν ∈ V0

  • and there exists an optimal control ˆ

ν0 of the auxiliary problem case 1. µ0 = ¯ f : then it is optimal to “do nothing” case 2. µ0 < ¯ f : one can rewrite the HJB equation ˜ w(x) = min ˜ ψ(x), ˜ ℓ(x) + P ˜ w(x)

  • with ˜

w = w0 − h(x, 0), ˜ ψ = Mw0 − h(x, 0), ˜ ℓ(x) = (¯ f − µ0)Ex0τ1. This is the HJB equation of a discrete optimal stopping problem which has an optimal control ˆ η = inf

  • n ≥ 0 : ˜

w(xn) = ˜ ψ(xn)

  • , i.e., ˆ

η = inf

  • n ≥ 0 : w0(xn) = Mw0(xn)
  • where

xn is the Markov chain xτn. From this, we deduce an optimal control with θ1 = τˆ

η, and θi = θˆ ηi with ˆ

ηi = inf

  • n ≥ ˆ

ηi : w0(xn) = Mw0(xn)

  • .

MR (UP-S) Ergodic Impulse Control with Constraint 13 / 19

slide-14
SLIDE 14

Solution Existence of an Optimal Control (3)

Existence of an Optimal Control (3) Corollary µ0 = inf ˜ J(x, y, ν), v ∈ V

  • and the optimal control ˆ

ν obtained by translation by τ1 of the control ˆ ν0. The final result is given by Theorem µ0 = inf{J(x, y, ν) : ν ∈ V} = J(x, y, ˆ ν)

MR (UP-S) Ergodic Impulse Control with Constraint 14 / 19

slide-15
SLIDE 15

Solution Existence of an Optimal Control (4)

Existence of an Optimal Control (4) A first step is to prove: Proposition (µ0, w0) being the solution previously obtained and recalling that w(x, y) = Ex0 τ1 [f − µ0]ds + w0(xτ1, 0)

  • (µ0, w) is solution of

−Axyw(x, y) + λ(x, y)[w(x, 0) − Mw(x, 0)]+ = f − µ0. where Axy is the (weak) infinitesimal generator of the uncontrolled process Axyϕ = Axϕ + ∂ϕ ∂y + λ(x, y)

  • ϕ(x, 0) − ϕ(x, y)
  • .

MR (UP-S) Ergodic Impulse Control with Constraint 15 / 19

slide-16
SLIDE 16

Solution Existence of an Optimal Control (5)

Existence of an Optimal Control (5) To prove the proposition, one first shows w0(x) = min{w(x, 0), Mw(x, 0)} which gives w(x, y) = Exy τ1 [f − µ0]dt + w(xτ1, 0) − [w(xτ1, 0) − Mw]+ from which we deduce that equation.✷ Next, the proposition allows us to show that MT = T [f (xt, yt) − µ0]ds + w(xT, yT) is a submartingale. This gives µ0 ≤ J(x, y, ν), ∀ ν ∈ V, and, from the first expression of w(x, y), on obtains µ0 = J(x, y, ν).

MR (UP-S) Ergodic Impulse Control with Constraint 16 / 19

slide-17
SLIDE 17

Solution Existence of an Optimal Control (5)

Extensions E locally compact.

  • If we assume Φ(t)C0(E) ⊂ C0(E) all other assumptions being

unchanged, one can still obtain the results on the HJB equation.

  • However, the “additional assumption” is no longer sufficient to get the

result on the existence of an optimal cost.

  • adding “h(x, 0) bounded” would be sufficient (e.g., if −Axyh = f − ¯

f has a bounded solution), but more general assumptions would require further work.

  • When λ is independent of x and f independent of y, one can obtain an
  • ptimal control without an additional assumption.

MR (UP-S) Ergodic Impulse Control with Constraint 17 / 19

slide-18
SLIDE 18

Solution Existence of an Optimal Control (5)

Extensions (2)

  • P(x, B) ≥ m(B) is also restrictive for E locally compact

Replacing by a “localized” condition like P(x, B) ≥ α✶K(x)m(B), allows to obtains the results on the HJB equations and, if h(x, 0) is bounded, the existence of an optimal control.

MR (UP-S) Ergodic Impulse Control with Constraint 18 / 19

slide-19
SLIDE 19

Solution References (among others)

References (among others) Besides the classical works on impulse control,

  • A. Bensoussan and J.L. Lions (book 1982), etc, . . . ,
  • For discrete time control: A. Bensoussan (book, 2011),
  • O. Hernandez-Lerma and J. Lasserre (books 1996, 1999), . . .
  • For average control, ergodicity: Bensoussan (book 1988),
  • M. Kurano (paper 1989),
  • F. Luque-Vasquez and O. Hernandez-Lerma (paper 1999),
  • L. Stettner (paper 1986), D. Gatarek and L. Stettner (paper 1990),
  • A. Arapostathis et al. (survey paper 1993),

M.G. Garroni and J.L. Menaldi (book 2002), Several papers with J.L. Menaldi (1997, 2013, . . . ), Papers to appear with J.L. Menaldi, H. Jasso-Fuentes, T. Prieto-Rumeau.

MR (UP-S) Ergodic Impulse Control with Constraint 19 / 19