Occupation measures and LMI formulation of piecewise affine optimal - - PowerPoint PPT Presentation

Occupation measures and LMI formulation of piecewise affine optimal control design problems Didier Henrion1,2 Luis Rodrigues3

1LAAS-CNRS Univ Toulouse, FR 2Czech Tech Univ Prague, CZ 3Concordia Univ Montr´

eal, CA Louvain-la-Neuve, 27 April 2010

Outline

• 1. Occupation measures and dynamical systems
• 2. LMI formulation of moment problem
• 3. Illustration: LQ optimal control
• 4. Application: PWA optimal control

Measures Measure = function assigning a volume or probability to a set µ : X ⊂ Rn → R+ µ(X) =

• X dµ =
• X dµ(x) =
• X µ(dx)

Examples:

• Lebesgue measure dµ(x) = dx, µ(X) = vol(X)
• Hermite measure dµ(x) = e−xT xdx (density, abs. continuous)
• probability measure µ(Rn) = 1
• Dirac measure dµ(x) = δx∗, µ({x∗}) = 1 (atomic, discrete)

Measures Support = smallest closed set X ⊂ Rn for which µ(Rn/X) = 0 Examples:

• Dirac measure supp(δx) = {x}
• atomic measure supp(µ) = {x1, . . . , xr}
• Hermite measure supp(µ) = Rn
• Lesbegue measure on X = [−1, 1], vol(X) = 2
• Lesbegue measure on X = {x ∈ R2 : xTx ≤ 1}, vol(X) = π

Indicator, or characteristic function of a set X IX(x) = 1 if x ∈ X =

• therwise

Moments Multi-index notation xα = n

i=1 xαi i

with x ∈ Rn, α ∈ Nn The α-th moment of measure µ is the real number yα =

• X xαdµ(x)

µ is a representing measure for sequence y = (yα)α∈Nn Classical problem of moments (Hausdorff, Markov, Stieltjes): characterise sequence y having representing measure µ supported on a (given) set X Conditions on yα ? Construction of µ and X, given y ?

Occupation measures Dynamical system described by ODE ˙ x = f(x) with solution, or flow φt(x0) starting from initial condition x0 Occupation measure of trajectory from φ0(x0) = x0 to φT(x0) = xT µ(X) =

T

0 IX(φt)dt

where X is a subset of Rn It is the time spent in X by the solution of the ODE

Moments of occupation measure By definition, in any subset X ⊂ Rn, the α-th moment of occupation measure µ is given by yα =

• X xαdµ(x)

=

• X

T

0 xαδx(φt)dtdx

=

T

• X xαδx(φt)dxdt

=

T

0 φα t dt

So if sequence y is given we can find system trajectories by solving the corresponding problem of moments Given dynamics f(x), how can we find sequence y ?

Measure problem Consider a continuously differentiable test function v(x) whose time-derivative along system trajectories is given by ˙ v = ∇v · ˙ x = ∇v · f From the fundamental theorem of calculus

T

0 dv

=

T

0 ∇v · fdt

=

• X ∇v · fdµ

= v(xT) − v(x0) it follows that occupation measure µ satisfies (infinitely many) linear equations

• X ∇v · fdµ = v(xT) − v(x0)

∀v

Measure problem Now assume x0 and xT are not known exactly, they are modeled by probability measures µ0 and µT with supports X0 and XT, respectively Our three measures satisfy the following constraints

• X ∇v · fdµ =
• XT

vdµT −

• X0

vdµ0, ∀v (compare with previous slide where µ0 = δx0, µT = δxT ) This is an infinite-dimensional linear problem in measure space Compare with weak or variational formulations of PDE problems Interesting mathematically, but not really constructive..

Polynomial optimal control Consider now the polynomial optimal control problem minu,T

T

0 h(x, u)dt + hT(x(T))

s.t. ˙ x = f(x) + g(x)u x(0) ∈ X0, x(T) ∈ XT x ∈ X, u ∈ U with f, g, h, hT polynomials and X0, XT, X, U compact basic semialgebraic sets (intersections of polynomial sublevel sets) Using the same ideas, this can be written as a linear but ∞-dimensional problem on measures minµ,µ0,µT

• X hdµ +
• XT hTdµT

s.t.

• X ∇v · (f(x) + g(x)u)dµ =
• X0 vdµ0 −
• XT vdµT, ∀v

Generalized problem of moments To summarize, we have a linear problem involving several measures µi respectively supported on sets Xi All the data are polynomials, so we can replace measures by their moments (e.g.

• Xi hi(x)dµi =
• Xi
• α hiαxαdµi =

α hiα

• Xi xαdµi)

minµ

• i
• Xi hidµi

s.t.

• i
• Xi aijdµi = bj

measures µi miny

• i
• α hiαyiα

s.t.

• i
• α aijαyiα = bj

moments yi provided we can handle the representation condition yiα =

• Xi

xαdµi(x)

Outline

• 1. Occupation measures and dynamical systems
• 2. LMI formulation of moment problem
• 3. Illustration: LQ optimal control
• 4. Application: PWA optimal control

LMI conditions Given a sequence y, define the moment matrix Md(y) of order d with entries indexed by multi-indices β (rows) and γ (columns) [Md(y)]β,γ = yβ+γ, |β| + |γ| ≤ 2d which are linear in y Necessary condition: if y has a representing measure µ then Md(y) 0 ∀d Sufficient condition (Berg 1987): if |yα| ≤ 1 ∀α and Md(y) 0 ∀d, then y has a representing measure µ with supp(µ) ⊂ [−1, 1]

LMI conditions Given a sequence y and a polynomial g(x) =

α gαxα, define the

localising matrix Md(gy) of order d with entries [Md(gy)]β,γ =

• α

gαyα+β+γ, |α| + |β| + |γ| ≤ 2d Let X = {x ∈ Rn : gi(x) ≥ 0, ∀i} be compact basic semialgebraic with {x : gi(x) ≥ 0} compact for some i Necessary condition: if y has a representing measure µ with support in X, then Md(y) 0, Md(giy) 0 ∀i ∀d Sufficient condition (Putinar 1993): if Md(y) 0, Md(giy) 0 ∀i ∀d then y has a representing measure with supp(µ) ⊂ X

Moment LP as LMI miny cTy s.t. Ay = b yα =

• X xαdµ

X = {x : gi(x) ≥ 0, ∀i} infinite-dimensional LP problem miny cTy s.t. Ay = b Md(y) 0 Md(giy) 0, ∀i finite-dim. LMI relaxation of order d GloptiPoly 3 (DH, JB. Lasserre, J. L¨

• fberg) for Matlab

models generalised problems of moments as LMI problems POCP (C. Savorgnan) models polynomial optimal control problems as generalised problems of moments homepages.laas.fr/henrion/software

Dual LMI Dual LMI problem yields polynomial subsolution h(x, u) + ∇v(x) · (f(x) + g(x)u) ≥ 0, v(x(T)) = hT(x(T))

• f Hamilton-Jacobi-Bellman PDE with positivity condition

relaxed to polynomial sum-of-squares LMI condition Good approximation of value function along optimal trajectories If h(x, u) = hx(x) + uTu, use optimality condition ∂(h(x, u) + ∇v(x) · (f(x) + g(x)u)) ∂u = 2u + ∇v · g = 0 to derive state-feedback control law u∗(x) = −1 2∇v∗(x) · g(x)

Outline

• 1. Occupation measures and dynamical systems
• 2. LMI formulation of moment problem
• 3. Illustration: LQ optimal control
• 4. Application: PWA optimal control

Linear ODE analysis Consider the scalar linear ODE ˙ x = −x with initial measure µ0 in X0 = {x : g0(x) = 1 − (x − 3

2)2 ≥ 0}

with final measure µT in XT = {x : gT(x) = 1

4 − x2 ≥ 0}

with occupation measure µ in X = {x : g(x) = 4 − x2 ≥ 0} We want to find trajectories minimising the energy

T

0 x2dt

Linear measure problem min

T

0 x2dµ(x)

s.t.

• X

dv(x) dx (−x)dµ(x) =

• XT vdµT −
• X0 vdµ0,

∀v

Linear ODE analysis Setting v = xα we introduce sequences y0, yT, y representing measures µ0, µT, µ, and we obtain the linear moment problem min y2 s.t. −αyα = yT α − y0α, ∀α and the corresponding LMI relaxation of order d min y2 s.t. −αyα = yT α − y0α, ∀α, |α| ≤ 2d Md(y0) 0, Md(yT) 0, Md(y) 0 Md(g0y0) 0, Md(gTyT) 0, Md(gy) 0 Solving LMI relaxations of increasing orders d yields a sequence

• f monotonically increasing lower bounds on the optimum

Linear ODE analysis This problem can be solved analytically, with optimal trajectory x(t) = e−t leaving X0 at x(0) = 1 and reaching XT at x(T) = 1

2

for T = log 2 ≈ 0.6931 Moment matrix M(y) has entries yα =

log 2

e−αtdt = 1−2−α

α

We get with SeDuMi 1.1R3 the following sequence of valid significant digits on T: 0, 2, 4, 7, 10, 13 (fast convergence) Convergence at a finite relaxation order is impossible since the

• ptimum is transcendental, whereas the solution of an integer

coefficient LMI is algebraic

LQR design Consider the linear quadratic regulator design problem minu,T

T

0 (x2 + u2)dt

s.t. ˙ x = u x(0) = x0, x(T) = xT with given initial and terminal conditions Measure µ0 is the Dirac δx0 and measure µT is the Dirac δxT so that only measure µ must be found Linear measure problem minµ

T

0 (x2 + u2)dµ(x, u)

s.t.

• R2 dv(x)

dx udµ(x, u) = v(xT) − v(x0),

∀v

LQR design Moment LMI problem min y20 + y02 s.t. y01 = 2y11 = 3y21 = · · · = −1 Md(y) 0 The moment matrix has the following quasi-Hankel structure Md(y) =

    

y00 y10 y01 y10 y20 y11 y01 y11 y02 ...

    

with yα =

• xα1uα2dµ(x, u)

LQR design Solving with SeDuMi 1.1R3 the LMI relaxation of order d = 1 yields M1(y) =

 

3.66 1.00 −1.00 1.00 0.50 −0.50 −1.00 −0.50 0.50

  Z =  

0.00 0.00 0.00 0.00 1.00 1.00 0.00 1.00 1.00

 

where Z is the multiplier matrix such that trace M1(y)Z = 0 From entries M1(y) we can read the optimal trajectory, with T ≈ 3.66 (exact value = ∞, but objective function almost equal),

x = 1, u = −1, x2 = 1

2 etc

LQR design From multipliers corresponding to equality constraints we retrieve v∗(x) = x2 as a polynomial subsolution to the HJB PDE, which is here a standard algebraic Riccati equation From Z we notice that the sum of the second (indexed by x) and third (indexed by u) row/column in M(y) vanishes, so that the optimal control policy u∗(x) satisfies the equation x + u∗(x) = 0 We can also use the optimality condition u∗(x) = −1 2∇v∗(x) = −x

Outline

• 1. Occupation measures and dynamical systems
• 2. LMI formulation of moment problem
• 3. Illustration: LQ optimal control
• 4. Application: PWA optimal control

Piecewise affine optimal control Optimal control problem minu,T

T

0 h(x, u)dt + hT(x(T))

s.t. ˙ x = Aix + ai + Biu when x ∈ Xi x(0) = x0, x(T) = xT where state-space is partitioned X = ∪iXi into compact basic semialgebraic sets Xi (e.g. polytopes) Multiple open-loop equilibrium points since ai = 0 We want a good approximation of the optimal control law u∗(x)

Several measures Introduce local occupation measures µi supported in each cell Xi Global occupation measure µ =

i µi

Linear optimal cost

• X hdµ +
• XT

hTdµT and linear constraints on measures

• X ∇v ·
• i

(Aix + ai + Biu)dµi =

• XT

vdµT −

• X0

vdµ0, ∀v and hence linear constraints on repective moment sequences yi

Example Consider the nonlinear system ˙ x = 1 2(1 − x2) + u with two equilibrium points, approximated globally by a piecewise affine system ˙ x = −x + 1 + u if x ≥ 0 = x + 1 + u if x ≤ 0 and we would like to solve the optimal control problem min

u,T

T

0 (2(1 − x)2 + u2)dt

with boundary conditions x(0) = −1, x(T) = +1

Optimal control From necessary optimality conditions on the piecewise affine Hamiltonian we obtain the analytic solution u∗(x) = (1 − √ 3)(x − 1) if x ≥ 0 = −x − 1 +

• 2(x − 1)2 + (x + 1)2

if x ≤ 0 showing in passing that an optimal controller for a PWA system is not necessarily PWA Using LMI relaxations of orders 2, 4, 6, . . . , 20 we obtain the following approximations to u∗(x)

−1 −0.8 −0.6 −0.4 −0.2 0.2 0.4 0.6 0.8 1 −0.5 0.5 1 1.5 2 2.5 3 x u(x)

Optimal feedback (red) and degree 2 approximation (black)

−1 −0.8 −0.6 −0.4 −0.2 0.2 0.4 0.6 0.8 1 −0.5 0.5 1 1.5 2 2.5 3 x u(x)

Optimal feedback (red) and degree 4 approximation (black)

−1 −0.8 −0.6 −0.4 −0.2 0.2 0.4 0.6 0.8 1 −0.5 0.5 1 1.5 2 2.5 3 x u(x)

Optimal feedback (red) and degree 6 approximation (black)

−1 −0.8 −0.6 −0.4 −0.2 0.2 0.4 0.6 0.8 1 −0.5 0.5 1 1.5 2 2.5 3 x u(x)

Optimal feedback (red) and degree 8 approximation (black)

−1 −0.8 −0.6 −0.4 −0.2 0.2 0.4 0.6 0.8 1 −0.5 0.5 1 1.5 2 2.5 3 x u(x)

Optimal feedback (red) and degree 10 approximation (black)

−1 −0.8 −0.6 −0.4 −0.2 0.2 0.4 0.6 0.8 1 −0.5 0.5 1 1.5 2 2.5 3 x u(x)

Optimal feedback (red) and degree 12 approximation (black)

−1 −0.8 −0.6 −0.4 −0.2 0.2 0.4 0.6 0.8 1 −0.5 0.5 1 1.5 2 2.5 3 x u(x)

Optimal feedback (red) and degree 14 approximation (black)

−1 −0.8 −0.6 −0.4 −0.2 0.2 0.4 0.6 0.8 1 −0.5 0.5 1 1.5 2 2.5 3 x u(x)

Optimal feedback (red) and degree 16 approximation (black)

−1 −0.8 −0.6 −0.4 −0.2 0.2 0.4 0.6 0.8 1 −0.5 0.5 1 1.5 2 2.5 3 x u(x)

Optimal feedback (red) and degree 18 approximation (black)

−1 −0.8 −0.6 −0.4 −0.2 0.2 0.4 0.6 0.8 1 −0.5 0.5 1 1.5 2 2.5 3 x u(x)

Optimal feedback (red) and degree 20 approximation (black)

Concluding remarks Occupation measures can handle piecewise affine models, but also much more (piecewise polynomial models, impulsive controls, hybrid dynamics) Hierarchy of LMI relaxations to generate polynomial approximations of optimal value function and optimal feedback Dynamical systems theory: Liouville’s theorem, advection PDEs Discrete-time piecewise affine systems: chaotic dynamics (e.g. tent map), invariant measures, ergodic theory, Frobenius-Perron and Koopmans operators