The HJB-POD approach for infinite dimensional control problems M. - - PowerPoint PPT Presentation

The HJB-POD approach for infinite dimensional control problems

• M. Falcone

works in collaboration with A. Alla, D. Kalise and S. Volkwein

Università di Roma “La Sapienza”

Numerical methods for Hamilton-Jacobi equations in optimal control and related fields RICAM, Linz, November 21, 2016

• M. Falcone (Università di Roma “La Sapienza”)

The HJB-POD approach 1 / 48

Outline

Outline

1

HJ equations, DP schemes and feedback synthesis HJ Equations Numerical scheme for HJ equation

2

Efficient numerical methods for HJ equations A quick overview Accelerated iterative schemes

3

HJB-POD method for high dimensional problem A-priori estimates for the HJB approximation A-priori estimates for the HJB-POD approximation

4

Numerical Tests

• M. Falcone (Università di Roma “La Sapienza”)

The HJB-POD approach 2 / 48

Outline

Introduction

The Dynamic Programming Principle allows to derive a first order partial differential equation describing the value function associated to the optimal control problem (in finite or infinite dimension). The theory of viscosity solutions allows to characterize the value function as the unique weak solution of the Bellman equation. This characterization has been used also to construct numerical schemes for the value function and to compute optimal feedbacks. Reference

• M. Bardi, I. Capuzzo Dolcetta, Optimal control and viscosity solutions
• f Hamilton-Jacobi-Bellman equations, 1997.
• M. Falcone (Università di Roma “La Sapienza”)

The HJB-POD approach 3 / 48

Outline

DP’s advantages and disadvantages

PROS

• 1. The characterization of the value function is valid for all classical

problems in any dimension.

• 2. The approximation is based on a-priori error estimates in L∞ and is

valid in any dimension

• 3. DP (semi-Lagrangian) schemes can work on structured and

unstructured grids.

• 4. The computation of feedbacks is almost built in and there are nice

results in low dimension.

• M. Falcone (Università di Roma “La Sapienza”)

The HJB-POD approach 4 / 48

Outline

DP’s advantages and disadvantages

PROS

• 1. The characterization of the value function is valid for all classical

problems in any dimension.

• 2. The approximation is based on a-priori error estimates in L∞ and is

valid in any dimension

• 3. DP (semi-Lagrangian) schemes can work on structured and

unstructured grids.

• 4. The computation of feedbacks is almost built in and there are nice

results in low dimension. CONS The "curse of dimensionality" makes the problem difficult to solve in high dimension due to

• 1. computational cost
• 2. huge memory allocations.
• M. Falcone (Università di Roma “La Sapienza”)

The HJB-POD approach 4 / 48

HJ equations, DP schemes and feedback synthesis

Outline

1

HJ equations, DP schemes and feedback synthesis HJ Equations Numerical scheme for HJ equation

2

Efficient numerical methods for HJ equations A quick overview Accelerated iterative schemes

3

HJB-POD method for high dimensional problem A-priori estimates for the HJB approximation A-priori estimates for the HJB-POD approximation

4

Numerical Tests

• M. Falcone (Università di Roma “La Sapienza”)

The HJB-POD approach 5 / 48

HJ equations, DP schemes and feedback synthesis HJ Equations

HJB equation for the infinite horizon problem

Controlled Dynamics and Cost Functional ˙ y(t) = f(y(t), u(t)), t ∈ (t0, +∞] y(t0) = x, Infinite horizon cost functional Jx(y, u) = +∞ g(y(s), u(s))e−λs ds Value Function v(x) := inf

u(·)∈U Jx(y, u(·)).

• M. Falcone (Università di Roma “La Sapienza”)

The HJB-POD approach 6 / 48

HJ equations, DP schemes and feedback synthesis Numerical scheme for HJ equation

Value and HJB equation: infinite horizon problem

Dynamic Programming Principle v(x) = min

u∈U

τ

t0

e−λsg(yx(s), u(s)) ds + v(yx(τ))e−λτ

• By Dynamic Programming we get the stationary Bellman equation

λv(x) + max

u∈U {−f(x, u) · ∇v(x) − g(x, u)} = 0,

x ∈ Rn Since the value function is in general not regular we need to use weak solutions, typically Lipschitz continuous. The value function is the unique viscosity solution of the Bellman equation. The construction of the approximation scheme can be obtained via a discrete dynamic programming approach. State constraints can also be included.

• M. Falcone (Università di Roma “La Sapienza”)

The HJB-POD approach 7 / 48

HJ equations, DP schemes and feedback synthesis Numerical scheme for HJ equation

Synthesis of feedback controls

The numerical solution of optimal control problems via HJB PDEs leads to the computation of feedback controls for generic nonlinear Lipschitz continuous vectorfields and costs. Solving λv(x) + max

u∈U {−f(x, u) · ∇v(x) − g(x, u)} = 0 ,

x ∈ Ω we get the value function on a grid and we extend it to the whole

• domain. Then, we can also compute a feedback map u∗ : Ω → U

u∗(x) ≡ arg min

u∈U{f(x, u) · ∇v(x) + g(x, u)} ,

x ∈ Ω which is used to compute optimal trajectories by an ODE scheme.

• M. Falcone (Università di Roma “La Sapienza”)

The HJB-POD approach 8 / 48

HJ equations, DP schemes and feedback synthesis Numerical scheme for HJ equation

The Zermelo navigation problem

1 0.8 0.6 0.4 0.2 0.2 0.4 0.6 0.8 1 1 0.8 0.6 0.4 0.2 0.2 0.4 0.6 0.8 1 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 1 0.8 0.6 0.4 0.2 0.2 0.4 0.6 0.8 1 1 0.8 0.6 0.4 0.2 0.2 0.4 0.6 0.8 1 1 2 3 4 5 6

Value Function Feedback

• M. Falcone (Università di Roma “La Sapienza”)

The HJB-POD approach 9 / 48

Efficient numerical methods for HJ equations

Outline

1

HJ equations, DP schemes and feedback synthesis HJ Equations Numerical scheme for HJ equation

2

Efficient numerical methods for HJ equations A quick overview Accelerated iterative schemes

3

HJB-POD method for high dimensional problem A-priori estimates for the HJB approximation A-priori estimates for the HJB-POD approximation

4

Numerical Tests

• M. Falcone (Università di Roma “La Sapienza”)

The HJB-POD approach 10 / 48

Efficient numerical methods for HJ equations A quick overview

How can we compute the value function?

The bottleneck of the DP approach is the computation of the value function, since this requires to solve a non linear PDE in high-dimension. This is a challenging problem due to the huge number of nodes involved and to the singularities of the solution. This goal has motivated new efforts in several directions: Domain Decomposition Fast Marching /Fast Sweeping Methods Accelerated iterative schemes High-order and adaptive grid methods

• M. Falcone (Università di Roma “La Sapienza”)

The HJB-POD approach 11 / 48

Efficient numerical methods for HJ equations Accelerated iterative schemes

Semi-Lagrangian discretization of HJB

Dynamic Programming Principle v(x) = min

u∈U

τ

t0

e−λsg(yx(s), u(s)) ds + v(yx(τ))e−λτ

• Time-Discrete Approximation via Value Iteration

V k+1

i

= min

u∈U{e−λ∆tV k (xi + ∆t f (xi, u)) + ∆t g (xi, u)}

Fix a grid in Ω with Ω ⊂ Rn bounded, Steps ∆x. Nodes: {x1, . . . , xN}, Discrete solution: Vi ≈ v(xi). Stability for large time steps ∆t

• M. Falcone (Università di Roma “La Sapienza”)

The HJB-POD approach 12 / 48

Efficient numerical methods for HJ equations Accelerated iterative schemes

Semi-Lagrangian discretization of HJB

The most standard way to solve this system is the Value Iteration (VI). Fully discrete SL-FEM/ Value Iteration (VI) scheme V k+1 = T(V k) , for i = 1, . . . , N

• T(V k)
• i

≡ min

u∈U{e−λ∆tI[V k](xi + ∆t f(xi, u)) + g(xi, u)}

Some advantages: Simple to implement (for I = I1,the P1 interpolation operator) (VI) Converges under very general assumptions. WARNING: Rather expensive in terms of CPU time, since β = e−λ∆t the Lipschitz constant of T goes to 1 when ∆t → 0.

• M. Falcone (Università di Roma “La Sapienza”)

The HJB-POD approach 13 / 48

Efficient numerical methods for HJ equations Accelerated iterative schemes

Semi-Lagrangian discretization of HJB

Fully-Discrete Approximation (Value Iteration) V k+1

i

= min

u∈U{e−λ∆tI

• V k

(xi + ∆t f (xi, u)) + ∆t L (xi, u)} This algorithm converges for any initial guess V 0. Error Estimate: [F . 1987] max

i∈NG

v(xi) − Vi ≤ C∆t1/2 + Lf λ(λ − Lf) ∆x ∆t . NG = number of nodes, Lf = Lipschitz constant of the dynamics f.

• M. Falcone (Università di Roma “La Sapienza”)

The HJB-POD approach 14 / 48

Efficient numerical methods for HJ equations Accelerated iterative schemes

Policy iteration

An alternative form to solve this problem is the iteration in the policy space [Bellman 1955, Howard 1960]. Fully discrete SL-FEM/ Policy Iteration (PI) scheme

1

Fix u0

i ∈ U, for i = 1, . . . , K.

2

Solve (V k)i = βI1[V k](xi + ∆t f(xi, uk

i )) + ∆t g(xi, uk i ).

3

Update uk+1

i

= argmin

u∈U

{I1[V k](xi + ∆t f(xi, u)) + ∆t g(xi, u)}.

4

Repeat until matching convergence criteria (can be set on V or u).Typically we use ||V k+1 − V k||∞ < ǫ as stopping rule. Note that in Step 2 the control is frozen.

• M. Falcone (Università di Roma “La Sapienza”)

The HJB-POD approach 15 / 48

Efficient numerical methods for HJ equations Accelerated iterative schemes

Numerical “Fact” #1: PI is faster

||V k − V ∗||∞ evolution for VI (2D MTP)

20 40 60 80 100 120 10

−2

10

−1

10 # of iterations L∞ error 212 DoF 412 DoF 812 DoF 1612 DoF

• M. Falcone (Università di Roma “La Sapienza”)

The HJB-POD approach 16 / 48

Efficient numerical methods for HJ equations Accelerated iterative schemes

Numerical “Fact” #1: PI is faster

||V k − V ∗||∞ evolution for PI (2D MTP)

10 20 30 40 10

−2

10

−1

10 # of iterations L∞ error 212 DoF 412 DoF 812 DoF 1612 DoF

• M. Falcone (Università di Roma “La Sapienza”)

The HJB-POD approach 17 / 48

Efficient numerical methods for HJ equations Accelerated iterative schemes

Numerical “Fact” #2: PI is sensitive to u0

i .

||V k − V ∗||∞ evolution for PI with different u0

i

5 10 15 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 # of iterations L∞ error Guess 1 Guess 2 Guess 3 Guess 4

• M. Falcone (Università di Roma “La Sapienza”)

The HJB-POD approach 18 / 48

Efficient numerical methods for HJ equations Accelerated iterative schemes

Numerical “Fact” #2: PI is sensitive to u0

i .

PI inner solver iteration subcount (2D MTP)

2 4 6 8 10 12 14 16 18 10 20 30 40 50 60 70 80

# of global iterations Subiterations count Guess 1 Guess 4 Guess 3

• M. Falcone (Università di Roma “La Sapienza”)

The HJB-POD approach 19 / 48

Efficient numerical methods for HJ equations Accelerated iterative schemes

• Num. “Fact” #3: VI is fast in coarse meshes.

||V k − V ∗||∞ evolution for VI (2D MTP)

20 40 60 80 100 120 10

−2

10

−1

10 # of iterations L∞ error 212 DoF 412 DoF 812 DoF 1612 DoF

• M. Falcone (Università di Roma “La Sapienza”)

The HJB-POD approach 20 / 48

Efficient numerical methods for HJ equations Accelerated iterative schemes

The accelerated algorithm (Alla, F., Kalise, 2015)

Theoretical results have established a link between VI and PI in control problems and games [Puterman and Brumelle (1979), Bokanowski, Maroso and Zidani (2009)]. Numerically, PI strongly depends on a good initialization. Similar approaches have been studied by González and Sagastizábal (1990), Chow (1991), Seeck (1997), and Grüne (2004). The Accelerated PI (API) algorithm

1

Consider two mesh parameters k1 and k2, k1 > k2. Set V 0

k1.

2

Perform a coarse VI in the k1− mesh with result V ∗

k1.

3

V 0

k2 = I1[V ∗ k1] and u0 k2 = argmin u∈U

{I1[V 0

k2](xi + hf(xi, u))}.

4

start PI solver over the k2-mesh.

• M. Falcone (Università di Roma “La Sapienza”)

The HJB-POD approach 21 / 48

Efficient numerical methods for HJ equations Accelerated iterative schemes

Test 1: different performances

L1-norm ||V k − V ∗||1 evolution.

10 20 30 40 50 60 70 80 90 10

−2

10

−1

10 CPU TIME L−inf NORM

2D EIKONAL EQUATION,16 CONTROLS, DELTAX=0.0125.

Value Iteration Policy Iteration Accelerated PI

• M. Falcone (Università di Roma “La Sapienza”)

The HJB-POD approach 22 / 48

HJB-POD method for high dimensional problem

Outline

1

HJ equations, DP schemes and feedback synthesis HJ Equations Numerical scheme for HJ equation

2

Efficient numerical methods for HJ equations A quick overview Accelerated iterative schemes

3

HJB-POD method for high dimensional problem A-priori estimates for the HJB approximation A-priori estimates for the HJB-POD approximation

4

Numerical Tests

• M. Falcone (Università di Roma “La Sapienza”)

The HJB-POD approach 23 / 48

HJB-POD method for high dimensional problem

Control of Partial Differential Equations via DP

The discretization of a PDE leads to a large system of ODEs. The approximation of the correspondent HJB equations becomes unfeasible because of the curse of dimensionality. Model Reduction via Proper Orthogonal Decomposition POD decomposition allows to reduce the number of variables to approximate partial differential equations. GOAL: to approximate PDE optimal control problems in a rather small dimension via POD using numerical schemes for HJB equations. References Kunisch and Volkwein (2001, ...), Kunisch, Volkwein and Xie (2004), Alla-F . (2013, 2014), Alla-Hinze (2015).

• M. Falcone (Università di Roma “La Sapienza”)

The HJB-POD approach 24 / 48

HJB-POD method for high dimensional problem

Proper Orthogonal Decomposition and SVD

Given snapshots : (y(t0, u), . . . , y(tm, u)) ∈ Rn We look for an orthonormal basis {ψi}ℓ

i=1 in Rm with ℓ ≤ min{n, m} s.t.

J(ψ1, . . . , ψℓ) =

m

• j=1

αj

• yj −

ℓ

• i=1

yj, ψiψi

• 2

=

d

• i=ℓ+1

σ2

i

reaches a minimum where {αj}n

j=1 ∈ R+.

min J(ψ1, . . . , ψℓ) s.t.ψi, ψj = δij Singular Value Decomposition: Y = ΨΣV T. For ℓ ∈ {1, . . . , d = rank(Y)}, {ψi}ℓ

i=1 is called the POD basis of rank ℓ.

Ansatz: y(x, t) ≈

ℓ

• i=1

yℓ

i (t)ψi(x)

• M. Falcone (Università di Roma “La Sapienza”)

The HJB-POD approach 25 / 48

HJB-POD method for high dimensional problem

Reduced Order Modeling Control Problem

min Jℓ(yℓ, u) yℓ satisfies the reduced dynamics ˙ yℓ(t) = F ℓ(yℓ(t), u(t)) t > 0, yℓ(t0) = yℓ

0 ∈ Rℓ.

The cost functional is: Jℓ

yℓ

0(yℓ, u) =

∞ gℓ(yℓ(t), u(t))e−λt dt Reduced Value Function v(yℓ

0) =

inf

u∈Uad

Jℓ(yℓ

yℓ

0, u)

• M. Falcone (Università di Roma “La Sapienza”)

The HJB-POD approach 26 / 48

HJB-POD method for high dimensional problem

Reduced Domain (in the POD space) for HJB

We are going to solve the HJB equation in a box: [a1, b1] × . . . × [aℓ, bℓ] The box must be chosen such that the dynamic contains all the possible controlled trajectories. We discretize the control space U by a discrete set {u1, . . . , uM}. y(t) =

ℓ

• i=1

< y(t, uj), ψi > ψi =

ℓ

• i=1

yℓ

i (t, uj)ψi

and we choose the box to guarantee yℓ

i (t, uj) ∈ [ai, bi], j = 1, . . . , M.

• M. Falcone (Università di Roma “La Sapienza”)

The HJB-POD approach 27 / 48

HJB-POD method for high dimensional problem A-priori estimates for the HJB approximation

Back to the infinite horizon problem

Let us consider the controlled dynamics ˙ y(t) = f

• y(t), u(t)
• ∈ Rn for t > 0,

y(0) = y◦ ∈ Rn (1) where f : Rn × Rm → Rn The infinite horizon cost functional is J(y, u) = ∞ g

• y(t), u(t)
• e−λt dt

(2) where λ > 0 and g : Rn × Rm → R. The set of admissible controls has the form Uad =

• u ∈ U
• u(t) ∈ Uad for almost all t ≥ 0
• ,

where U = L2(0, ∞; Rm) and Uad ⊂ Rm is a compact, convex subset.

• M. Falcone (Università di Roma “La Sapienza”)

The HJB-POD approach 28 / 48

HJB-POD method for high dimensional problem A-priori estimates for the HJB approximation

Variational formulation of the dynamics

Let M ∈ Rn×n denote a symmetric, positive definite (mass) matrix with smallest and largest positive eigenvalues λmin and λmax, respectively. We introduce the following weighted inner product in Rn: y, ˜ yM = y⊤M˜ y for y, y ∈ Rn, By · M = · , ·1/2

M

we define the associated induced norm. Recall that we have λmin y2

2 ≤ y2 M ≤ λmax y2 2

for all y ∈ Rn. Then, y is a trajectory if ˙ y(t) − f(y(t), u(t)), ϕM = 0 for all ϕ ∈ Rn and for almost all t > 0, y(0) − y◦, ϕM = 0 for all ϕ ∈ Rn (3)

• M. Falcone (Università di Roma “La Sapienza”)

The HJB-POD approach 29 / 48

HJB-POD method for high dimensional problem A-priori estimates for the HJB approximation

The infinite horizon problem

Assume that the dynamics has a unique solution y = y(u; y◦) ∈ Y = H1(0, ∞; Rn) for every admissible control u ∈ Uad and for every initial condition y◦ ∈ Rn, denoted by y(u; y◦) . Reduced cost functional

• J(u; y◦) = J(y(u; y◦), u)

for u ∈ Uad and y◦ ∈ Rn, Then, our optimal control problem for y◦ ∈ Rn is min

u∈Uad

• J(u; y◦).

( P)

• M. Falcone (Università di Roma “La Sapienza”)

The HJB-POD approach 30 / 48

HJB-POD method for high dimensional problem A-priori estimates for the HJB approximation

ASSUMPTIONS

(A1) f : Rn × Rm → Rn is continuous and f(y, u) − f(˜ y, u)2 ≤ Lf y − ˜ y2 for all y, ˜ y ∈ Rn and u ∈ Uad Moreover, f∞ ≤ Mf for (y, u) ∈ Ω × Uad. (A2) g : Rn × Rm → Rn is continuous and globally Lipschitz continuous. Moreover, g(y, u)∞ ≤ Mg for (y, u) ∈ Ω × Uad. If (A1)–(A2) hold and λ > Lf, vh is globally Lipschitz-continuous

• vh(y) − vh(˜

y)

• ≤

Lg λ − Lf y − ˜ y2 ∀y, ˜ y ∈ Ω and h ∈ [0, 1/λ)

• M. Falcone (Università di Roma “La Sapienza”)

The HJB-POD approach 31 / 48

HJB-POD method for high dimensional problem A-priori estimates for the HJB approximation

A-priori estimate for HJB [F87]

Theorem Let (A1)–(A2) hold and λ > max{Lg, Lf}. Let v and vh be the continuous and semi-discrete solutions respectively. Moreover, assume semiconcavity, i.e. f(y + ˜ y, u) − 2f(y, u) + f(y − ˜ y, u)2 ≤ Cf ˜ y2

2,

• g(y + ˜

y, u) − 2g(y, u) + g(y − ˜ y, u)

• ≤ Cg ˜

y2

2

(4) for all (y, ˜ y, u) ∈ Rn × Rn × Uad. Then, sup

y∈Rn

• v(y) − vh(y)
• ≤ Ch

for any h ∈ [0, 1/λ).

• M. Falcone (Università di Roma “La Sapienza”)

The HJB-POD approach 32 / 48

HJB-POD method for high dimensional problem A-priori estimates for the HJB-POD approximation

An estimate for the HJB-POD approximation

We introduce two different POD approximations for the HJB equation. A first estimate is based on the fully discrete HJB equation, where we project all vertices {yi}nS

i=1 into Rℓ by setting

yℓ

i = Ψ⊤Myi

for i = 1, . . . , nS. Here we assume yℓ

i = yℓ j for i, j ∈ {1, . . . , nS} with i = j.

Then, a POD discretization of HJB in the reduced space is given by vℓ

hk(yℓ i ) = min u∈Uad

• (1 − λh)vℓ

hk

• yℓ

i + hf ℓ(yℓ i , u)

• + hgℓ(yℓ

i , u)

• (5)

for 1 ≤ i ≤ nS.

• M. Falcone (Università di Roma “La Sapienza”)

The HJB-POD approach 33 / 48

HJB-POD method for high dimensional problem A-priori estimates for the HJB-POD approximation

An estimate for the HJB-POD approximation

Let us define the mapping vℓ

hk : Ω → R by

• vℓ

hk(y) = vℓ hk(Ψ⊤My)

for all y ∈ Ω with Ψ⊤My ∈ Ω. Thus, (5) can be written as

• vℓ

hk(yi) = min u∈Uad

• (1 − λh)

vℓ

hk

• yi + hf(Pℓyi, u)
• + hg(Pℓyi, u)
• (6)

for 1 ≤ i ≤ nS.

• M. Falcone (Università di Roma “La Sapienza”)

The HJB-POD approach 34 / 48

HJB-POD method for high dimensional problem A-priori estimates for the HJB-POD approximation

An estimate for the HJB-POD approximation

To simplify, let us assume the invariance condition for the reduced order dynamics yi + hf(Pℓyi, u) ∈ Ω for i = 1, . . . , nS, ∀u ∈ Uad (A3) Theorem Assume that (A1)–(A2), (A3) and λ > Lf hold. Then, there exist two constants C0, C1 such that sup

y∈Ω

• vh(y) −

vℓ

hk(y)

• ≤

C0 k h + C1 nS

• i=1

yi − Pℓyi

2 2

1/2 for any h ∈ [0, 1/λ)

• M. Falcone (Università di Roma “La Sapienza”)

The HJB-POD approach 35 / 48

HJB-POD method for high dimensional problem A-priori estimates for the HJB-POD approximation

An estimate for the HJB-POD approximation

By the previous results we conclude Theorem Assume that (A1)–(A2), (A3) and (A4) hold. Let f, g satisfy the semiconcavity conditions. If λ > max{Lf, Lg}, then there exists constants C0, C1, C2 ≥ 0 such that sup

y∈Ω

• v(y) −

vℓ

hk(y)

• ≤ C0h + C1

k h + C2 nS

• i=1

yi − Pℓyi

2 2

1/2 for any h ∈ [0, 1/λ).

• M. Falcone (Università di Roma “La Sapienza”)

The HJB-POD approach 36 / 48

Numerical Tests

Outline

1

HJ equations, DP schemes and feedback synthesis HJ Equations Numerical scheme for HJ equation

2

Efficient numerical methods for HJ equations A quick overview Accelerated iterative schemes

3

HJB-POD method for high dimensional problem A-priori estimates for the HJB approximation A-priori estimates for the HJB-POD approximation

4

Numerical Tests

• M. Falcone (Università di Roma “La Sapienza”)

The HJB-POD approach 37 / 48

Numerical Tests

Test 3: Advection-Diffusion Equation

Advection-Diffusion Equation    yt − εyxx + βyx = χΩc(x)u(t) in Ω × (0, ∞], y(·, 0) = y0, in Ω, y(·, t) = 0 in ∂Ω × (0, T), (7) Cost Functional J(y, u, t) = ∞

• y(x, τ)2 + γ||u(τ)||2

e−λτ dτ. Parameters ε = 0.1, β = 1, γ = 0.01, y0(x) = 0.5 sin(πx), Ω = [0, 2], Ωc = (0.5, 1), U = [−2.2, 0]; Snapshots: ∆x = 0.01, ∆t = 0.1. VF: ∆x = {0.1, 0.05}, ∆t = 0.1∆x; Trajectories: ∆t = 0.01;

• M. Falcone (Università di Roma “La Sapienza”)

The HJB-POD approach 38 / 48

Numerical Tests

Test 3: Advection-Diffusion Equation

0.5 1 1.5 2 1 2 3 0.1 0.2 0.3 0.4 0.5 time 0.5 1 1.5 2 1 2 3 −0.1 0.1 0.2 0.3 0.4 time 0.5 1 1.5 2 1 2 3 −0.1 0.1 0.2 0.3 0.4 time

Controls

0.5 1 1.5 2 2.5 3 −2.5 −2 −1.5 −1 −0.5 time 0.5 1 1.5 2 2.5 3 −2 −1.5 −1 −0.5 time

Figure: Test 3: Uncontrolled (left), LQR optimal (middle), HJB-POD (right).

• M. Falcone (Università di Roma “La Sapienza”)

The HJB-POD approach 39 / 48

Numerical Tests

Test 3: Advection-Diffusion Equation

2 3 4 0.021 0.0211 0.0212 0.0213 0.0214 0.0215 0.0216 Number of POD basis K=0.1 K=0.05 2 3 4 0.01 0.02 0.03 0.04 0.05 Number of POD basis k=0.1 k=0.05 2 3 4 0.005 0.01 0.015 0.02 0.025 0.03 Number of POD basis K=0.1 K=0.05

Figure: Evaluation of the cost functional (left), L2−error for y(uℓ) and yℓ(uℓ) (middle) and L2-error between LQR solution and y(uℓ).

• M. Falcone (Università di Roma “La Sapienza”)

The HJB-POD approach 40 / 48

Numerical Tests

Test 4: Semi-linear Equation

Semi-Linear Equation    yt − εyxx + βyx + µ(y − y3) = y0(x)u(t) in Ω × (0, ∞], y(·, 0) = y0, in Ω, y(·, t) = 0 in ∂Ω × (0, T), Cost Functional J(y, u, t) = ∞

• y(x, τ)2 + γ||u(τ)||2

e−λτ dτ. Parameters ε = 0.1 = β, µ = 1, γ = 0.01, y0(x) = sin(πx), Ω = [0, 1], U = [−1, 1]; Snapshots: ∆x = 0.01, ∆t = 0.1. VF: ∆x = {0.1, 0.05}, ∆t = 0.1∆x; Trajectories: ∆t = 0.01;

• M. Falcone (Università di Roma “La Sapienza”)

The HJB-POD approach 41 / 48

Numerical Tests

Test 4: Semi-linear Equation

0.2 0.4 0.6 0.8 1 1 2 3 −0.1 0.1 0.2 0.3 0.4 0.5 time 0.2 0.4 0.6 0.8 1 1 2 3 −0.1 0.1 0.2 0.3 0.4 0.5 time 0.2 0.4 0.6 0.8 1 1 2 3 −0.1 0.1 0.2 0.3 0.4 0.5 time 0.5 1 1.5 2 2.5 3 −1 −0.8 −0.6 −0.4 −0.2 0.2 0.4 0.6 0.8 1 time 0.5 1 1.5 2 2.5 3 −1 −0.8 −0.6 −0.4 −0.2 0.2 0.4 0.6 0.8 1 time

Figure: Uncontrolled (left), Optimal (middle) and optimal HJB control (right).

• M. Falcone (Università di Roma “La Sapienza”)

The HJB-POD approach 42 / 48

Numerical Tests

Test 5: Wave Equation

Damped Wave Equation    ytt − yxx − βyxxt = χΩc(x)u(t) in Ω × (0, ∞], y(·, 0) = y0, yt(·, 0) = y1 in Ω, y(·, t) = 0 in ∂Ω × (0, T), Cost Functional J(y, u, t) = ∞

• y(x, τ)2 + γ||u(τ)||2

e−λτ dτ. Parameters β = 0.05, γ = 0.01, y0(x) = sin(πx), y1(x) = 0, Ω = [0, 1], Ωc = (0.4, 0.6), U = [−1.2, 0.6]; Snapshots: ∆x = 0.01, ∆t = 0.1. VF: ∆t = 0.01, ∆x = 0.1; Trajectories: ∆t = 0.01;

• M. Falcone (Università di Roma “La Sapienza”)

The HJB-POD approach 43 / 48

Numerical Tests

Test 5: Wave Equation, Optimal Solution

0.2 0.4 0.6 0.8 1 1 2 3 4 −1 −0.5 0.5 1 TIME 0.2 0.4 0.6 0.8 1 1 2 3 4 −1 −0.5 0.5 1 TIME 0.2 0.4 0.6 0.8 1 1 2 3 4 −1 −0.5 0.5 1 TIME

k = 0.1 k = 0.05 ℓ = 3 0.1224 0.0953 ℓ = 4 0.1009 0.0886 ℓ = 5 0.0648 0.0468

Table: H1

0−error between LQR control and HJB-POD approx. at time t = 4.

• M. Falcone (Università di Roma “La Sapienza”)

The HJB-POD approach 44 / 48

Numerical Tests

Test 5: Wave Equation, Stabilization of the feedback

0.5 1 1.5 2 2.5 3 3.5 4 −1 −0.8 −0.6 −0.4 −0.2 0.2 0.4 0.6 0.5 1 1.5 2 2.5 3 3.5 4 −1 −0.8 −0.6 −0.4 −0.2 0.2 0.4 0.6

Figure: Feedback Control with Chattering (left) LQR control (right)),

• M. Falcone (Università di Roma “La Sapienza”)

The HJB-POD approach 45 / 48

Numerical Tests

Wave Equation: Stabilization of the feedback

0.5 1 1.5 2 2.5 3 3.5 4 −1 −0.8 −0.6 −0.4 −0.2 0.2 0.4 0.6 0.5 1 1.5 2 2.5 3 3.5 4 −1 −0.8 −0.6 −0.4 −0.2 0.2 0.4 0.6

Figure: Feedback Control with Chattering (left), Stabilized feedback control with limited chattering (right)

• M. Falcone (Università di Roma “La Sapienza”)

The HJB-POD approach 46 / 48

Numerical Tests

CONCLUSIONS Accelerated Policy Iteration speeds up the numerical approximation of the value function Model reduction via POD is crucial in order to make the problem feasible Feedback Control for PDEs is obtained via the POD-HJB approach The chattering of the feedback control for finite and infinite dimensional problem can be reduced A-priori error estimation are available

• M. Falcone (Università di Roma “La Sapienza”)

The HJB-POD approach 47 / 48

Numerical Tests

References for this talk

• A. Alla, M. Falcone An adaptive POD approximation method for the control of

advection-diffusions equation, 2013.

• A. Alla, M. Falcone, D. Kalise, An efficient Policy Iteration algorithm for Dynamic

Programming equations, 2015.

• A. Alla, M. Falcone, D. Kalise, A HJB-POD feedback synthesis approach for the

wave equation, 2015.

• A. Alla, M. Falcone, S. Volkwein, Error Analysis for POD approximations of

infinite horizon problems via the dynamic programming principle, submitted 2015.

• M. Falcone, R. Ferretti, Semi-Lagrangian approximation schemes for linear and

Hamilton–Jacobi equations , SIAM, 2014.

• M. Falcone, R. Ferretti, Numerical methods for Hamilton–Jacobi type equations,

in Handbook of Numerical Methods for Hyperbolic Problems, Springer, 2016

• K. Kunisch, S. Volkwein, L. Xie, HJB-POD Based Feedback Design for the

Optimal Control of Evolution Problems, 2004.

• S. Volkwein. Model Reduction using Proper Orthogonal Decomposition, Lecture

Notes, 2013.

• M. Falcone (Università di Roma “La Sapienza”)

The HJB-POD approach 48 / 48