Optimal Control and Hamilton-Jacobi Equations H el` ene - - PowerPoint PPT Presentation

Value Function HJB Equation State Constraints

Optimal Control and Hamilton-Jacobi Equations

H´ el` ene Frankowska

CNRS and UNIVERSIT´ E PIERRE et MARIE CURIE

Control and Optimization, Monastir, Tunisia May 15-19, 2017

• H. Frankowska

Optimal Control

Value Function HJB Equation State Constraints

Deterministic Control System

• ˙

x(t) = f (t, x(t), u(t)), u(t) ∈ U a.e. in [0, 1] (CS) x(t0) = x0 U is a complete separable metric space, t denotes the time f : [0, 1] × Rn × U → Rn, x0 ∈ Rn Controls are Lebesgue measurable functions u(·) : [0, 1] → U A trajectory of (CS) is any absolutely continuous function x ∈ W 1,1([t0, 1]; Rn) satisfying for some control u(·) ˙ x(t) = f (t, x(t), u(t)) a.e. in [0,1] We assume that f (t, x, ·) is continuous f (·, x, u) is measurable, f (t, x, U) are closed and ∃ γ : [0, 1] → R+ integrable such that supu∈U |f (t, x, u)| ≤ γ(t)(1 + |x|)

• H. Frankowska

Optimal Control

Value Function HJB Equation State Constraints

Deterministic Control System

• ˙

x(t) = f (t, x(t), u(t)), u(t) ∈ U a.e. in [0, 1] (CS) x(t0) = x0 U is a complete separable metric space, t denotes the time f : [0, 1] × Rn × U → Rn, x0 ∈ Rn Controls are Lebesgue measurable functions u(·) : [0, 1] → U A trajectory of (CS) is any absolutely continuous function x ∈ W 1,1([t0, 1]; Rn) satisfying for some control u(·) ˙ x(t) = f (t, x(t), u(t)) a.e. in [0,1] We assume that f (t, x, ·) is continuous f (·, x, u) is measurable, f (t, x, U) are closed and ∃ γ : [0, 1] → R+ integrable such that supu∈U |f (t, x, u)| ≤ γ(t)(1 + |x|)

• H. Frankowska

Optimal Control

Value Function HJB Equation State Constraints

Deterministic Control System

• ˙

x(t) = f (t, x(t), u(t)), u(t) ∈ U a.e. in [0, 1] (CS) x(t0) = x0 U is a complete separable metric space, t denotes the time f : [0, 1] × Rn × U → Rn, x0 ∈ Rn Controls are Lebesgue measurable functions u(·) : [0, 1] → U A trajectory of (CS) is any absolutely continuous function x ∈ W 1,1([t0, 1]; Rn) satisfying for some control u(·) ˙ x(t) = f (t, x(t), u(t)) a.e. in [0,1] We assume that f (t, x, ·) is continuous f (·, x, u) is measurable, f (t, x, U) are closed and ∃ γ : [0, 1] → R+ integrable such that supu∈U |f (t, x, u)| ≤ γ(t)(1 + |x|)

• H. Frankowska

Optimal Control

Value Function HJB Equation State Constraints

Semilinear Control System

X is a separable Banach space. Consider the densely defined unbounded linear operator A - the infinitesimal generator of a strongly continuous semigroup S(t) : X → X, f : [0, 1] × X × U → X, x0 ∈ X and the semilinear control system ˙ x(t) = Ax + f (t, x(t), u(t)), u(t) ∈ U, x(t0) = x0 Its mild trajectory is defined by x(t) = S(t − t0) x0 +

t

t0

S(t − s) f (s, x(s), u(s)) ds ∀ t ∈ [t0, 1] Many of the deterministic results were already adapted to the framework of semilinear control systems (controlled PDEs). Naturally this means some assumptions on semigroups: S(·) has to be compact to prove existence of optimal controls.

• H. Frankowska

Optimal Control

Value Function HJB Equation State Constraints

Optimal Control

The concept of optimal control can be described as the process

• f influencing the behavior of a dynamical system so as to

achieve the desired goal: to maximize a profit, to minimize the energy, to get from one point to another one, etc. “After correctly stating the problem of optimal control and having at hand some satisfactory existence theorems, augmented by necessary conditions for optimality, we can consider that we have sufficiently substantial basis to study some special problems, as for instance Moon Flight Problem”. From a book on Optimal Control, 1969

• H. Frankowska

Optimal Control

Value Function HJB Equation State Constraints

Value Function of the Mayer Problem

g : Rn → R ∪ {+∞}, ξ0 ∈ Rn. Consider the Mayer’s problem: min {g(x(1)) | x is a trajectory of (CS), x(0) = ξ0} The value function associated with this problem is defined by: ∀ (t0, x0) ∈ [0, 1] × Rn V (t0, x0) = inf{g(x(1)) | x is a trajectory of (CS), x(t0) = x0} V (1, ·) = g(·). In general V is nonsmooth even for smooth data. If x(·) is a trajectory of (CS), then for any t0 ≤ t1 ≤ t2 ≤ 1, V (t1, x(t1)) ≤ V (t2, x(t2)). ¯ x(·) is optimal if and only if V (t, ¯ x(t)) ≡ g(¯ x(1)). Dynamic Programming Principle: ∀ h > 0 such that t0 + h ≤ 1 V (t0, x0) = inf{V (t+h, x(t+h)) | x is a trajectory of (CS), x(t0) = x0}

• H. Frankowska

Optimal Control

Value Function HJB Equation State Constraints

Hamilton-Jacobi Equation

The Hamiltonian H : [0, 1] × Rn × Rn → R is defined by H(t, x, p) = max

u∈U p, f (t, x, u)

If V ∈ C1 it satisfies the Hamilton-Jacobi equation −Vt(t, x) + H (t, x, −Vx(t, x)) = 0, V (1, ·) = g(·) Optimal (feedback) control u(t, x) ∈ U is chosen by : −Vx(t, x), f (t, x, u(t, x)) = H (t, x, −Vx(t, x)) If u(t, ·) is Lipschitz, then the solution ¯ x(·) of ˙ x(t) = f (t, x(t), u(t, x)) a.e. in [0, 1], x(0) = ξ0 is optimal for the Mayer problem. Even if data are smooth, V is not differentiable.

• H. Frankowska

Optimal Control

Value Function HJB Equation State Constraints

Active Mathematical Domains Motivated by Optimal Control

Nonsmooth, Set-Valued Analysis, Variational Analysis (since 1975) Solutions of HJB equations : viscosity and bilateral (since 1983) Control under state constraints (since 2000) First order necessary optimality conditions (since 1957) Second order necessary optimality conditions (since 1965) Sensitivity relations in control (since 1986) .....

• H. Frankowska

Optimal Control

Value Function HJB Equation State Constraints

Outline

1

Value Function of the Mayer Problem

2

Hamilton-Jacobi-Bellman Equation

3

State Constraints

• H. Frankowska

Optimal Control

Value Function HJB Equation State Constraints

Existence of Optimal Controls

Below we always assume that for a.e. t and ∀ r > 0, f (t, ·, u) is cr(t)-Lipschitz on B(0, r) ∀ u ∈ U, with integrable cr : [0, 1] → R. For the Mayer problem let (xi, ui) be a minimizing sequence of trajectory-control pairs. By Gronwall’s Lemma, {xi∞} is bounded and |˙ xi(t)| ≤ γ(t)(xi∞ + 1). Take a subsequence {xij} converging uniformly to some ¯ x : [0, 1] → Rn with ˙ xij converging weakly in L1([0, 1]; Rn) to some ¯ y : [0, 1] → Rn. Then ˙ ¯ x = ¯ y, ¯ x(0) = ξ0 and ˙ ¯ x(t) ∈ conv f (t, ¯ x(t), U) a.e. where conv denotes convex hull. If f (t, x, U) is convex, then, by the measurable selection theorem, there exists a control ¯ u such that ˙ ¯ x(t) = f (t, ¯ x(t), ¯ u(t)) a.e.

• H. Frankowska

Optimal Control

Value Function HJB Equation State Constraints

Existence of Optimal Controls

If g is lower semicontinuous, then lim inf

j→∞ g(xij(1)) ≥ g(¯

x(1)) and therefore ¯ u is optimal control. Theorem If f (t, x, U) are convex and g is lower semicontinuous, then for the Mayer problem an optimal solution does exist. For semilinear control systems only a part of this proof applies and

• ne has either to assume, that S(t) is compact for all t > 0 or that

{(x, f (t, x, u)) | u ∈ U, x ∈ X} is convex. This last assumption is very strong. For nonlinear stochastic control systems the convexity assumptions are worse even in the finite dimensional framework.

• H. Frankowska

Optimal Control

Value Function HJB Equation State Constraints

Relaxation Theorem

˙ x(t) ∈ conv f (t, x(t), U) a.e. in [t0, 1] (RS) has more W 1,1([t0, 1]; Rn) solutions (relaxed trajectories) than the control system (CS) for the same initial condition x(t0) = x0. Theorem Let ¯ x : [t0, 1] → Rn be a relaxed trajectory. Then for every ε > 0 there exists a trajectory x of (CS) such that x − ¯ x∞ ≤ ε and x(t0) = ¯ x(t0). Corollary If g is continuous, then the infimum in the Mayer problem is attained by a relaxed trajectory and V = V rel, where V rel(t0, x0) = min{g(x(1)) | x ∈ W 1,1 satisfies (RS), x(t0) = x0}

• H. Frankowska

Optimal Control

Value Function HJB Equation State Constraints

Regularity of Value Function

Under our assumptions, sets of solutions of (CS) and (RS) depend

• n initial state in a locally Lipschitz way. Hence

Theorem If ∀ x ∈ Rn, f (t, x, U) is convex and g is lower semicontinuous, then V is lower semicontinuous. If g is continuous, then V is continuous. If g is locally Lipschitz, then V (t, ·) is locally Lipschitz for every t ∈ [0, 1]. Furthermore if γ is essentially bounded, then V is locally Lipschitz. If g ∈ C1, f (t, ·, u) ∈ C1, H(t, ·, ·) ∈ C2 on Rn × (Rn\{0}), ∇g never vanishes and for every initial datum the optimal solution of the relaxed problem is unique, then V ∈ C1.

• H. Frankowska

Optimal Control

Value Function HJB Equation State Constraints

Generalized Differentials of Nonsmooth Functions

Let ϕ : Rn → R ∪ {±∞} and x0 ∈ Rn be such that ϕ(x0) = ±∞. The Fr´ echet superdifferential of ϕ at x0 is the closed convex set ∂+ϕ(x0) = {p ∈ Rn| lim sup

x→x0

ϕ(x) − ϕ(x0) − p, x − x0 |x − x0| ≤ 0} The Fr´ echet subdifferential of ϕ at x0 is the set: ∂−ϕ(x0) = {p ∈ Rn| lim inf

x→x0

ϕ(x) − ϕ(x0) − p, x − x0 |x − x0| ≥ 0} We always have ∂+ϕ(x0) = −∂−(−ϕ)(x0).

• H. Frankowska

Optimal Control

Value Function HJB Equation State Constraints

Directional Derivatives of Nonsmooth Functions

The contingent epiderivative of ϕ at x0 in the direction v is D↑ϕ(x0)(v) = lim inf

ε→0+, v′→v

ϕ(x0 + εv′) − ϕ(x0) ε and the contingent hypoderivative of ϕ at x0 in the direction v D↓ϕ(x0)(v) = lim sup

ε→0+, v′→v

ϕ(x0 + εv′) − ϕ(x0) ε We always have ∂−ϕ(x0) = {p ∈ Rn | D↑ϕ(x0)(v) ≥ p, v ∀ v ∈ Rn} ∂+ϕ(x0) = {p ∈ Rn | D↓ϕ(x0)(v) ≤ p, v ∀ v ∈ Rn}

• H. Frankowska

Optimal Control

Value Function HJB Equation State Constraints

Sufficient Conditions for Optimality

Theorem Assume that g is locally Lipschitz, 0 ≤ t0 < 1. Consider a trajectory z of (CS). If for a.e. t ∈ [t0, 1], ∃ p(t) ∈ Rn such that (p(t), ˙ z(t) , −p(t)) ∈ ∂+V (t, z(t)) then z is optimal at (t0, z(t0)). Proof — ψ(t) := V (t, z(t)) is absolutely continuous. Let t ∈ [t0, 1] be such that ˙ ψ(t) and ˙ z(t) do exist and our assumption is verified. Then ˙ ψ(t) ≥ 0 and 0 = (p(t), ˙ z(t), −p(t)), (1, ˙ z(t)) ≥ D↓V (t, z(t))(1, ˙ z(t)) ≥ lim supε→0+ V (t + ε, z(t + ε)) − V (t, z(t)) ε = ˙ ψ(t) Thus ˙ ψ(t) = 0 and therefore t → V (t, z(t)) is constant.

• H. Frankowska

Optimal Control

Value Function HJB Equation State Constraints

Contingent Inequalities

Theorem Assume g is lsc, f is continuous in time uniformly in u and f (t, x, U) are convex. Then for all (t0, x0) ∈ Dom(V ),

    

(i) t0 < 1 = ⇒ infu∈U D↑V (t0, x0)(1, f (t0, x0, u)) ≤ 0 (ii) t0 > 0 = ⇒ supu∈U D↑V (t0, x0)(−1, −f (t0, x0, u)) ≤ 0 (iii) t0 < 1 = ⇒ infu∈U D↓V (t0, x0)(1, f (t0, x0, u)) ≥ 0 Proof — Consider a trajectory x of (CS) such that x(t0) = x0 V (t, x(t)) ≡ g(x(1)) and εi → 0+ such that x(t0+εi)−x(t0)

εi

→ v. Then, by convexity, v ∈ f (t0, x0, U). Therefore D↑V (t0, x0)(1, v) ≤ 0. Fix u0 ∈ U and consider the solution of (CS) with constant control u0. Then for ε > 0 V (t0−ε, x(t0−ε))−V (t0, x0) ≤ 0 ≤ V (t0+ε, x(t0+ε))−V (t0, x0). Divide by ε and take lower and upper limits.

• H. Frankowska

Optimal Control

Value Function HJB Equation State Constraints

Notion of Solution to (HJB)

Corollary (Bilateral Solution) LSC value function satisfies V (1, ·) = g, ∀ (t, x) ∈ ]0, 1[×Rn,

    

∀ (pt, px) ∈ ∂−V (t, x), −pt + H(t, x, −px) = 0 ∀ x ∈ Rn, V (0, x) = lim inft→0+, x→x V (t, x) ∀ x ∈ Rn, V (1, x) = lim inft→1−, x→x V (t, x) Corollary (Viscosity Solution) Continuous value function satisfies V (1, ·) = g, ∀ t ∈ ]0, 1[, x

• ∀ (pt, px) ∈ ∂−V (t, x), −pt + H(t, x, −px) ≥ 0

∀ (t, pt, px) ∈ ∂+V (t, x), −pt + H(x, −px) ≤ 0 If a function W satisfies conditions of either corollary, then it is equal to the value function.

• H. Frankowska

Optimal Control

Value Function HJB Equation State Constraints

Uniqueness of Solution to (HJB)

Steps of the proof of uniqueness Deduce from inequalities involving Fr´ echet super/sub differentials contingent inequalities. This step is not simple because D↑V (t0, x0)(·, ·) and D↓V (t0, x0)(·, ·) are not convex

• functions. It requires lots of nonsmooth analysis results.

Show that any function W satisfying contingent inequalities is nondecreasing along trajectories of (CS). Use invariance theorems. Show that for any function W satisfying contingent inequalities and any (t0, x0) ∈ dom(W ), there exists a trajectory x(·) of (CS) such that W (t, x(t)) is nonincreasing. Use the viability theorem. These two monotonicity properties imply W = V

• H. Frankowska

Optimal Control

Value Function HJB Equation State Constraints

State Constraints

For a given closed set K ⊂ Rn, state constraints are expressed by x(t) ∈ K for all t ∈ [0, 1] A trajectory x(·) as above is called a viable (or feasible) trajectory

• f the control system.

SK(x0) denotes set of all viable trajectories defined on [0, 1] starting at x0 at time 0. f (a0, u1), f (b0, u2), f (c0, u2) are tangent to K at a0, b0 and c0. At d0 there is no u ∈ U such that f (d0, u) is tangent to K.

• H. Frankowska

Optimal Control

Value Function HJB Equation State Constraints

Example : Milyutin, 2000 (also for higher order)

        

min

T

• x(t) + |u(t)|p

p

• dt, p > 1

x′′′(t) = u(t), x(0) = ξ0, x′(0) = ξ1, x′′(0) = ξ2 u(t) ∈ R, x(t) ≥ 0 If an initial condition (ξ0, ξ1, ξ2) is admissible, then the optimal solution exists and is unique. For “most” of the admissible initial conditions (ξ0, ξ1, ξ2) there exist T = T(ξ0, ξ1, ξ2) > 0 such that the optimal trajectory reaches the boundary of constraints an infinite (countable) number

• f times with T being their accumulation point.
• H. Frankowska

Optimal Control

Value Function HJB Equation State Constraints

Neighboring Feasible Trajectories (NFT) Estimates

Distance from z ∈ Rn to K, dK(z) := min{|z − y| : y ∈ K}. Can we “control” trajectories violating state constraints ? What are sufficient conditions for the following property : For every trajectory ˆ x (·) of (CS) with ˆ x (0) ∈ K there exists a viable trajectory x (·) ∈ SK(ˆ x (0)) satisfying the following NFT estimates in · W 1,1 x − ˆ xW 1,1 ≤ C max

t∈[0,1] dK (ˆ

x (t))

• r the NFT estimates in · ∞

x − ˆ x∞ ≤ C max

t∈[0,1] dK (ˆ

x (t)) where C depends on the magnitude of |ˆ x(0)|, but not on ˆ x (·) ?

• H. Frankowska

Optimal Control

Value Function HJB Equation State Constraints

Neighboring Feasible Trajectories (NFT) Estimates

Distance from z ∈ Rn to K, dK(z) := min{|z − y| : y ∈ K}. Can we “control” trajectories violating state constraints ? What are sufficient conditions for the following property : For every trajectory ˆ x (·) of (CS) with ˆ x (0) ∈ K there exists a viable trajectory x (·) ∈ SK(ˆ x (0)) satisfying the following NFT estimates in · W 1,1 x − ˆ xW 1,1 ≤ C max

t∈[0,1] dK (ˆ

x (t))

• r the NFT estimates in · ∞

x − ˆ x∞ ≤ C max

t∈[0,1] dK (ˆ

x (t)) where C depends on the magnitude of |ˆ x(0)|, but not on ˆ x (·) ?

• H. Frankowska

Optimal Control

Value Function HJB Equation State Constraints

Neighboring Feasible Trajectories (NFT) Estimates

Distance from z ∈ Rn to K, dK(z) := min{|z − y| : y ∈ K}. Can we “control” trajectories violating state constraints ? What are sufficient conditions for the following property : For every trajectory ˆ x (·) of (CS) with ˆ x (0) ∈ K there exists a viable trajectory x (·) ∈ SK(ˆ x (0)) satisfying the following NFT estimates in · W 1,1 x − ˆ xW 1,1 ≤ C max

t∈[0,1] dK (ˆ

x (t))

• r the NFT estimates in · ∞

x − ˆ x∞ ≤ C max

t∈[0,1] dK (ˆ

x (t)) where C depends on the magnitude of |ˆ x(0)|, but not on ˆ x (·) ?

• H. Frankowska

Optimal Control

Value Function HJB Equation State Constraints

Neighboring Feasible Trajectories (NFT) Estimates

Distance from z ∈ Rn to K, dK(z) := min{|z − y| : y ∈ K}. Can we “control” trajectories violating state constraints ? What are sufficient conditions for the following property : For every trajectory ˆ x (·) of (CS) with ˆ x (0) ∈ K there exists a viable trajectory x (·) ∈ SK(ˆ x (0)) satisfying the following NFT estimates in · W 1,1 x − ˆ xW 1,1 ≤ C max

t∈[0,1] dK (ˆ

x (t))

• r the NFT estimates in · ∞

x − ˆ x∞ ≤ C max

t∈[0,1] dK (ˆ

x (t)) where C depends on the magnitude of |ˆ x(0)|, but not on ˆ x (·) ?

• H. Frankowska

Optimal Control

Value Function HJB Equation State Constraints

• H. Frankowska

Optimal Control

Value Function HJB Equation State Constraints

K with C 1,1 Boundary

Let U be compact, f be time independent, and ∃ k ≥ 0 such that f (·, u) is k-Lipschitz (uniformly in u). Let K be the closure of an open set with C1,1 boundary ∂K and assume the inward pointing condition (Soner, 1986) : ∀ x ∈ ∂K min

u∈Unx, f (x, u) < 0,

where nx is the outward unit normal to K at x Then NFT estimates in · ∞ hold true. Can be extended to f measurable in time, by adding an inward pointing condition on a neighborhood of ∂K.

• H. Frankowska

Optimal Control

Value Function HJB Equation State Constraints

K with C 1,1 Boundary

Let U be compact, f be time independent, and ∃ k ≥ 0 such that f (·, u) is k-Lipschitz (uniformly in u). Let K be the closure of an open set with C1,1 boundary ∂K and assume the inward pointing condition (Soner, 1986) : ∀ x ∈ ∂K min

u∈Unx, f (x, u) < 0,

where nx is the outward unit normal to K at x Then NFT estimates in · ∞ hold true. Can be extended to f measurable in time, by adding an inward pointing condition on a neighborhood of ∂K.

• H. Frankowska

Optimal Control

Value Function HJB Equation State Constraints

K with C 1,1 Boundary

Let U be compact, f be time independent, and ∃ k ≥ 0 such that f (·, u) is k-Lipschitz (uniformly in u). Let K be the closure of an open set with C1,1 boundary ∂K and assume the inward pointing condition (Soner, 1986) : ∀ x ∈ ∂K min

u∈Unx, f (x, u) < 0,

where nx is the outward unit normal to K at x Then NFT estimates in · ∞ hold true. Can be extended to f measurable in time, by adding an inward pointing condition on a neighborhood of ∂K.

• H. Frankowska

Optimal Control

Value Function HJB Equation State Constraints

Tangent and Normal Cones

Let (X, d) be a metric space Aτ ⊂ X, τ ∈ T . Limsupτ→τ0Aτ = {v ∈ X | lim inf

τ→τ0 dAτ (v) = 0}

Liminfτ→τ0Aτ = {v ∈ X | lim sup

τ→τ0

dAτ (v) = 0} (the Peano-Kuratowski upper and lower limits) The contingent cone TK(x) to K ⊂ Rn at x ∈ K is the set of v ∈ Rn such that ∃ εi → 0+, vi → v satisfying x + εivi ∈ K TK(x) = Limsupε→0+ K − x ε The limiting normal cone to a closed subset K ⊂ Rn at x ∈ K is NL

K(x) = Limsupy→Kx[TK(y)]−

→K stands for the convergence in K and TK(y)− = {p ∈ Rn | p, v ≤ 0 ∀ v ∈ TK(y)}

• H. Frankowska

Optimal Control

Value Function HJB Equation State Constraints

General Set K

The Clarke normal cone to K at x ∈ K NK(x) := conv NL

K(x)

Normalized normals : N1

K(x) := NK(x) ∩ Sn−1

Generalized Inward Pointing Condition (HF, Rampazzo, 2000) min

u∈U

max

p∈N1

K(x)p, f (t, x, u) < 0 ∀ x ∈ ∂K

Assume f (·, ·, u) is locally Lipschitz (uniformly in u). Then NFT estimates in · ∞ hold true. A counterexample to NFT estimates in · W 1,1 was given in P. Bettiol, A. Bressan & R. Vinter, SICON (2010) for f independent from t, x and K being an intersection of two half spaces in R2,

• H. Frankowska

Optimal Control

Value Function HJB Equation State Constraints

General Set K

The Clarke normal cone to K at x ∈ K NK(x) := conv NL

K(x)

Normalized normals : N1

K(x) := NK(x) ∩ Sn−1

Generalized Inward Pointing Condition (HF, Rampazzo, 2000) min

u∈U

max

p∈N1

K(x)p, f (t, x, u) < 0 ∀ x ∈ ∂K

Assume f (·, ·, u) is locally Lipschitz (uniformly in u). Then NFT estimates in · ∞ hold true. A counterexample to NFT estimates in · W 1,1 was given in P. Bettiol, A. Bressan & R. Vinter, SICON (2010) for f independent from t, x and K being an intersection of two half spaces in R2,

• H. Frankowska

Optimal Control

Value Function HJB Equation State Constraints

General Set K

The Clarke normal cone to K at x ∈ K NK(x) := conv NL

K(x)

Normalized normals : N1

K(x) := NK(x) ∩ Sn−1

Generalized Inward Pointing Condition (HF, Rampazzo, 2000) min

u∈U

max

p∈N1

K(x)p, f (t, x, u) < 0 ∀ x ∈ ∂K

Assume f (·, ·, u) is locally Lipschitz (uniformly in u). Then NFT estimates in · ∞ hold true. A counterexample to NFT estimates in · W 1,1 was given in P. Bettiol, A. Bressan & R. Vinter, SICON (2010) for f independent from t, x and K being an intersection of two half spaces in R2,

• H. Frankowska

Optimal Control

Value Function HJB Equation State Constraints

General Set K

The Clarke normal cone to K at x ∈ K NK(x) := conv NL

K(x)

Normalized normals : N1

K(x) := NK(x) ∩ Sn−1

Generalized Inward Pointing Condition (HF, Rampazzo, 2000) min

u∈U

max

p∈N1

K(x)p, f (t, x, u) < 0 ∀ x ∈ ∂K

Assume f (·, ·, u) is locally Lipschitz (uniformly in u). Then NFT estimates in · ∞ hold true. A counterexample to NFT estimates in · W 1,1 was given in P. Bettiol, A. Bressan & R. Vinter, SICON (2010) for f independent from t, x and K being an intersection of two half spaces in R2,

• H. Frankowska

Optimal Control

Value Function HJB Equation State Constraints

Other Counterexamples

and a counterexample to NFT estimates in · ∞ when K is an intersection of two half spaces in R3 and f is independent from x and is measurable in time. May we expect a weaker logarithmic estimate when maxt∈[0,1] dK(ˆ x(t)) > 0 ? x − ˆ x∞ ≤ C max

t∈[0,1] dK(ˆ

x(t))

• log( max

t∈[0,1] dK(ˆ

x(t)))

• r the H¨
• lder estimate : for some α ∈ (0, 1)

x − ˆ x∞ ≤ C max

t∈[0,1] dK (ˆ

x (t))α There are a counterexample to the logarithmic estimates for f independent from x and continuous in time and a counterexample to H¨

• lder estimates for f independent from x

and measurable in time.

• H. Frankowska

Optimal Control

Value Function HJB Equation State Constraints

Other Counterexamples

and a counterexample to NFT estimates in · ∞ when K is an intersection of two half spaces in R3 and f is independent from x and is measurable in time. May we expect a weaker logarithmic estimate when maxt∈[0,1] dK(ˆ x(t)) > 0 ? x − ˆ x∞ ≤ C max

t∈[0,1] dK(ˆ

x(t))

• log( max

t∈[0,1] dK(ˆ

x(t)))

• r the H¨
• lder estimate : for some α ∈ (0, 1)

x − ˆ x∞ ≤ C max

t∈[0,1] dK (ˆ

x (t))α There are a counterexample to the logarithmic estimates for f independent from x and continuous in time and a counterexample to H¨

• lder estimates for f independent from x

and measurable in time.

• H. Frankowska

Optimal Control

Value Function HJB Equation State Constraints

Counterexample (Bettiol, Bressan, Vinter)

f (t, x, U) = G := co {(2, 1), (−2, 1), (0, 0)} K = {(x1, x2) : x2 ≥ |x1|}

• H. Frankowska

Optimal Control

Value Function HJB Equation State Constraints

Trajectory Violating State Constraints

ˆ x(t) = (ˆ x1(t), t)

• H. Frankowska

Optimal Control

Value Function HJB Equation State Constraints

Getting Rid of this Counterexample

G1 = G ∪ {(2, 3), (−2, 3)} Then NFT estimates in W 1,1 hold true

• H. Frankowska

Optimal Control

Value Function HJB Equation State Constraints

Revisiting Classical Condition

Let K be the closure of an open set with C2 boundary ∂K and assume the inward pointing condition : ∀ x ∈ ∂K, ∃ ux ∈ U with nx, f (x, ux) < 0. Then for every u ∈ U with nx, f (x, u) ≥ 0 nx, f (x, ux) − f (x, u) < 0 The generalization of this last inward pointing condition is : ∀ t ∈ [0, 1] , ∀ x ∈ ∂K, (IPC)

        

∀ v ∈ U with maxn∈N1

K(x)n, f (t, x, v) ≥ 0

∃ w ∈ Liminf(s,y)→(t,x)f (s, y, U) maxn∈N1

K(x)n, w − f (t, x, v) < 0

The relaxed inward pointing condition : ∀ t ∈ [0, 1] , ∀ x ∈ ∂K, (IPCrel) same as (IPC) but w ∈ Liminf(s,y)→(t,x) conv f (s, y, U)

• H. Frankowska

Optimal Control

Value Function HJB Equation State Constraints

Revisiting Classical Condition

Let K be the closure of an open set with C2 boundary ∂K and assume the inward pointing condition : ∀ x ∈ ∂K, ∃ ux ∈ U with nx, f (x, ux) < 0. Then for every u ∈ U with nx, f (x, u) ≥ 0 nx, f (x, ux) − f (x, u) < 0 The generalization of this last inward pointing condition is : ∀ t ∈ [0, 1] , ∀ x ∈ ∂K, (IPC)

        

∀ v ∈ U with maxn∈N1

K(x)n, f (t, x, v) ≥ 0

∃ w ∈ Liminf(s,y)→(t,x)f (s, y, U) maxn∈N1

K(x)n, w − f (t, x, v) < 0

The relaxed inward pointing condition : ∀ t ∈ [0, 1] , ∀ x ∈ ∂K, (IPCrel) same as (IPC) but w ∈ Liminf(s,y)→(t,x) conv f (s, y, U)

• H. Frankowska

Optimal Control

Value Function HJB Equation State Constraints

Revisiting Classical Condition

Let K be the closure of an open set with C2 boundary ∂K and assume the inward pointing condition : ∀ x ∈ ∂K, ∃ ux ∈ U with nx, f (x, ux) < 0. Then for every u ∈ U with nx, f (x, u) ≥ 0 nx, f (x, ux) − f (x, u) < 0 The generalization of this last inward pointing condition is : ∀ t ∈ [0, 1] , ∀ x ∈ ∂K, (IPC)

        

∀ v ∈ U with maxn∈N1

K(x)n, f (t, x, v) ≥ 0

∃ w ∈ Liminf(s,y)→(t,x)f (s, y, U) maxn∈N1

K(x)n, w − f (t, x, v) < 0

The relaxed inward pointing condition : ∀ t ∈ [0, 1] , ∀ x ∈ ∂K, (IPCrel) same as (IPC) but w ∈ Liminf(s,y)→(t,x) conv f (s, y, U)

• H. Frankowska

Optimal Control

Value Function HJB Equation State Constraints

For f (·, ·, U) having a Closed Graph

Theorem (NODEA, 2013) Assume that f (·, ·, U) has closed graph and (IPC). Then NFT estimates in · W 1,1 hold true. If f is measurable in time, then a similar result is valid but a uniform inward pointing condition has to be imposed on a neighborhood of the boundary of K. Theorem (NODEA, 2013) Assume that f (·, ·, U) has closed graph and (IPCrel). Then for any viable relaxed trajectory xrel(·) and every δ > 0 there exists a trajectory x(·) of (CS) such that x(0) = xrel(0), x(t) ∈ Int K ∀ t ∈ (0, 1] and x(·) − ˆ x(·)∞ ≤ δ

• H. Frankowska

Optimal Control

Value Function HJB Equation State Constraints

For f (·, ·, U) having a Closed Graph

Theorem (NODEA, 2013) Assume that f (·, ·, U) has closed graph and (IPC). Then NFT estimates in · W 1,1 hold true. If f is measurable in time, then a similar result is valid but a uniform inward pointing condition has to be imposed on a neighborhood of the boundary of K. Theorem (NODEA, 2013) Assume that f (·, ·, U) has closed graph and (IPCrel). Then for any viable relaxed trajectory xrel(·) and every δ > 0 there exists a trajectory x(·) of (CS) such that x(0) = xrel(0), x(t) ∈ Int K ∀ t ∈ (0, 1] and x(·) − ˆ x(·)∞ ≤ δ

• H. Frankowska

Optimal Control

Value Function HJB Equation State Constraints

For f (·, ·, U) having a Closed Graph

Theorem (NODEA, 2013) Assume that f (·, ·, U) has closed graph and (IPC). Then NFT estimates in · W 1,1 hold true. If f is measurable in time, then a similar result is valid but a uniform inward pointing condition has to be imposed on a neighborhood of the boundary of K. Theorem (NODEA, 2013) Assume that f (·, ·, U) has closed graph and (IPCrel). Then for any viable relaxed trajectory xrel(·) and every δ > 0 there exists a trajectory x(·) of (CS) such that x(0) = xrel(0), x(t) ∈ Int K ∀ t ∈ (0, 1] and x(·) − ˆ x(·)∞ ≤ δ

• H. Frankowska

Optimal Control

Value Function HJB Equation State Constraints

Relaxed Semilinear Control System

˙ x(t) ∈ Ax + conv f (t, x(t), U), x(t0) = x0 Its mild trajectory is defined by x(t) = S(t − t0) x0 +

t

t0

S(t − s) v(s) ds ∀ t ∈ [t0, 1] where v(s) ∈ conv f (s, x(s), U). Assume X is Hilbert and K = Int K ⊂ X. Denote by Z the set of points z ∈ X\∂K admitting a unique projection P∂K(z) on ∂K. For every z ∈ Z, set nz = z − P∂K(z) z − P∂K(z)X sgn(doriented

∂K

(z)) .

• H. Frankowska

Optimal Control

Value Function HJB Equation State Constraints

Relaxation of Semilinear Control Systems under State Constraint, MCRF, 2016

The analogue of relaxation theorem is valid under the following inward pointing condition: ∀ ¯ x ∈ ∂K, ∃ η, ρ, M > 0 such that ∀ t ∈ [0, 1], ∀ x ∈ K ∩ B(¯ x, η), ∀ v ∈ U satisfying sup

τ≤η, z∈Z∩B(x,η)

nz, S(τ) f (t, x, v) ≥ 0, we have

• ¯

v ∈ U : f (t, x, ¯ v) − f (t, x, v) ≤ M , sup

τ≤η, z∈Z∩B(S(τ)x,η)

nz, S(τ) (f (t, x, ¯ v) − f (t, x, v))

< −ρ

• = ∅ .

If K is convex, then Z can be replaced by X\K.

• H. Frankowska

Optimal Control

Value Function HJB Equation State Constraints

Relaxation of Semilinear Control Systems under State Constraint, MCRF, 2016

The analogue of relaxation theorem is valid under the following inward pointing condition: ∀ ¯ x ∈ ∂K, ∃ η, ρ, M > 0 such that ∀ t ∈ [0, 1], ∀ x ∈ K ∩ B(¯ x, η), ∀ v ∈ U satisfying sup

τ≤η, z∈Z∩B(x,η)

nz, S(τ) f (t, x, v) ≥ 0, we have

• ¯

v ∈ U : f (t, x, ¯ v) − f (t, x, v) ≤ M , sup

τ≤η, z∈Z∩B(S(τ)x,η)

nz, S(τ) (f (t, x, ¯ v) − f (t, x, v))

< −ρ

• = ∅ .

If K is convex, then Z can be replaced by X\K.

• H. Frankowska

Optimal Control

Value Function HJB Equation State Constraints

Value Function of the Constrained Mayer Problem

For g : Rn → R {+∞} the Mayer problem under state constraints is minimize {g(x(1)) | x(·) ∈ SK(ξ0)} The value function is defined by V (t0, x0) = inf{g(x(1)) | x(·) is a viable trajectory of (CS), x(t0) = x0} The relaxed Mayer problem is : min {g(x(1)) | x(·) is a viable relaxed trajectory of (RS), x(0) = ξ0}

• H. Frankowska

Optimal Control

Value Function HJB Equation State Constraints

Value Function of the Constrained Mayer Problem

For g : Rn → R {+∞} the Mayer problem under state constraints is minimize {g(x(1)) | x(·) ∈ SK(ξ0)} The value function is defined by V (t0, x0) = inf{g(x(1)) | x(·) is a viable trajectory of (CS), x(t0) = x0} The relaxed Mayer problem is : min {g(x(1)) | x(·) is a viable relaxed trajectory of (RS), x(0) = ξ0}

• H. Frankowska

Optimal Control

Value Function HJB Equation State Constraints

Lipschitz Continuity of the Value Function

Corollary (NODEA, 2013) Assume (IPCrel), f is continuous in t, uniformly in u, γ is essentially bounded and that g is locally Lipschitz. Then V is locally Lipschitz on [0, 1] × K and is equal to the value function of the relaxed problem. Furthermore if ¯ x(·) is an optimal solution to the Mayer problem, then it is also optimal for the relaxed problem. Outward Pointing Condition (OPC)

  

∀ t ∈ [0, 1] , x ∈ ∂K, v ∈ U with minn∈N1

K(x) n, f (t, x, v) ≤ 0

∃ u ∈ U, minn∈N1

K(x)∩Sn−1 n, f (t, x, u) − f (t, x, v) ≥ 0.

• H. Frankowska

Optimal Control

Value Function HJB Equation State Constraints

Lipschitz Continuity of the Value Function

Corollary (NODEA, 2013) Assume (IPCrel), f is continuous in t, uniformly in u, γ is essentially bounded and that g is locally Lipschitz. Then V is locally Lipschitz on [0, 1] × K and is equal to the value function of the relaxed problem. Furthermore if ¯ x(·) is an optimal solution to the Mayer problem, then it is also optimal for the relaxed problem. Outward Pointing Condition (OPC)

  

∀ t ∈ [0, 1] , x ∈ ∂K, v ∈ U with minn∈N1

K(x) n, f (t, x, v) ≤ 0

∃ u ∈ U, minn∈N1

K(x)∩Sn−1 n, f (t, x, u) − f (t, x, v) ≥ 0.

• H. Frankowska

Optimal Control

Value Function HJB Equation State Constraints

Constrained HJB Equation under (IPC)

Theorem (Uniqueness of constrained viscosity solutions,

• Calc. Var.& PDEs, 2013)

Let W : [0, 1] × K → R. Assume (IPC) and that g and H are continuous and W (1, ·) = g(·). Then the following statements are equivalent: (a) W is the value function of the Mayer problem; (b) W is continuous and (i) −pt + H(t, x, −px) ≤ 0, ∀ (pt, px) ∈ ∂+W (t, x), ∀ (t, x) ∈]0, 1[×Int K ; (ii) −pt + H(t, x, −px) ≥ 0, ∀ (pt, px) ∈ ∂−W (t, x), ∀ (t, x) ∈]0, 1[×K . In other words, V is the unique continuous viscosity solution of the constrained HJB equation.

• H. Frankowska

Optimal Control

Value Function HJB Equation State Constraints

Constrained HJB Equation under (OPC)

Theorem (Uniqueness of constrained bilateral solutions, Calc. Var.& PDEs, 2013) Assume (OPC), that H is continuous, g is lsc and let W : [0, 1] × K → R ∪ {+∞}, W (1, ·) = g(·). If f (t, x, U) are convex, then the following two statements are equivalent: (a) W is the value function of the Mayer problem; (b) W is lower semicontinuous and (i) −pt + H(t, x, −px) = 0, ∀ (pt, px) ∈ ∂−W (t, x), ∀ (t, x) ∈]0, 1[×Int K; (ii) −pt + H(t, x, −px) ≥ 0, ∀ (pt, px) ∈ ∂−W (t, x), ∀ (t, x) ∈ (]0, 1[×∂K) ∪ ({0} × K); (iii) lim inf (t′, x′) → (1, x) (t′, x′) ∈]0, 1[×Int K W (t′, x′) = g(x), ∀ x ∈ K.

• H. Frankowska

Optimal Control

Value Function HJB Equation State Constraints

Conclusions and Open Questions

Value function is the unique solution of Hamilton-Jacobi-Bellman equation Same in the presence of state constraints For the controlled PDEs that can be reduced to semilinear control systems without state constraints HJB were already investigated Many Open Questions in : HJB associated to semilinear control systems in the presence of state constraints Many Open Questions in : Stochastic Optimal Control NFT theorems for semilinear control systems - to appear in ESAIM: Control, Optimisation and Calculus of Variations

• H. Frankowska

Optimal Control