Solving High Dimensional Hamilton- Jacobi-Bellman Equations Using - - PowerPoint PPT Presentation

▶
solving high dimensional hamilton jacobi bellman
SMART_READER_LITE
LIVE PREVIEW

Solving High Dimensional Hamilton- Jacobi-Bellman Equations Using - - PowerPoint PPT Presentation

Sep 11, 2022 43 likes •439 views

Solving High Dimensional Hamilton- Jacobi-Bellman Equations Using Low Rank Tensor Decomposition Yoke Peng Leong California Institute of Technology Joint work with Elis Stefansson, Matanya Horowitz, Joel Burdick 1 Motivation Synthesize

slide-1
SLIDE 1

Solving High Dimensional Hamilton- Jacobi-Bellman Equations Using Low Rank Tensor Decomposition

Yoke Peng Leong California Institute of Technology

Joint work with Elis Stefansson, Matanya Horowitz, Joel Burdick

slide-2
SLIDE 2

Motivation

Synthesize optimal feedback controllers for nonlinear dynamical systems in high dimensions

1 > 40 degree of freedoms > 12 degree of freedoms

slide-3
SLIDE 3

Outline

  • Motivation
  • Problem formulation
  • Low rank tensor decomposition
  • Alternating least squares & improvements
  • Example
  • Summary & future work

2

slide-4
SLIDE 4

Problem Formulation

Stochastic nonlinear affine system: Synthesize a controller, u(x), that minimize the cost function:

Brownian noise 3

slide-5
SLIDE 5

HJB Equation

Dynamic programming gives the HJB equation where the optimal controller is given by Value function (cost-to-go):

PDE Nonlinear

4

slide-6
SLIDE 6

Log transformation:

Linearly Solvable HJB Equation

Condition:

HJB equation (a nonlinear PDE)

Any system of the following form can satisfy the condition

Linearly solvable HJB equation (a linear PDE)

Desirability function

  • W. H. Fleming, C. J. Holland, P. Dai Pra, R. Filliger, H. Kappen, E. Todorov,
  • F. Stulp, E. A. Theodorou, K. Dvijotham, S. Schaal

5

slide-7
SLIDE 7

Convex Optimization

Suboptimal Stabilizing Controllers for Linearly Solvable Systems, Y. P. Leong, M. B. Horowitz, J. W. Burdick, CDC 2015 Linearly Solvable Stochastic Control Lyapunov Functions, Y. P. Leong, M. B. Horowitz, J. W. Burdick, SIAM Journal on Control and Optimization, Accepted

Upper bound solution Relaxation Sum of squares program

6

slide-8
SLIDE 8

Main Results

Theorem: The approximate solution is a stochastic control Lyapunov function (SCLF). Corollary: The suboptimal controller is stabilizing in probability.

Proof: Relaxed HJB and satisfies the definition of SCLF.

The approximate solution gives a suboptimal stabilizing controller. Theorem: Given the controller , then

Expected cost of a system using the given controller

The approximate solution gives an upper bound to the actual cost when using the suboptimal controller.

Proof: Manipulate the relaxed HJB and the error bound of approximate value function. 7

slide-9
SLIDE 9

Convex Optimization

Suboptimal Stabilizing Controllers for Linearly Solvable Systems, Y. P. Leong, M. B. Horowitz, J. W. Burdick, CDC 2015 Linearly Solvable Stochastic Control Lyapunov Functions, Y. P. Leong, M. B. Horowitz, J. W. Burdick, SIAM Journal on Control and Optimization, Accepted

Upper bound solution Relaxation Sum of squares program

Problem: Curse of dimensionality

8

slide-10
SLIDE 10

Related Works

  • Sparse grid approximation (J. Garcke, A. Kröner)
  • Taylor polynomial approximation + patchy technique (C.
  • O. Aguilar, A. J. Krener)
  • Max-plus expansion (W. M. McEneaney)
  • Model reduction (K. Kunisch, S. Volkwein, L. Xie, S.

Gombao)

  • Level-set algorithm (I. M. Mitchell, C. J. Tomlin)

etc…

9

slide-11
SLIDE 11

Goal

Solve linear HJB equations for high dimensional dynamical systems

10 > 40 degree of freedoms > 12 degree of freedoms

slide-12
SLIDE 12

Outline

  • Motivation
  • Problem formulation
  • Low rank tensor decomposition
  • Alternating least squares & improvements
  • Example
  • Summary & future work

11

slide-13
SLIDE 13

Low Rank Tensor Decomposition

Low rank tensor decomposition Separated representation

  • G. Beylkin
  • M. J. Mohlenkamp

12

slide-14
SLIDE 14

Low Rank Tensor Decomposition

Low rank tensor decomposition

13

slide-15
SLIDE 15

Low Rank Tensor Decomposition

Low rank tensor decomposition CANDECOMP/PARAFAC tensor decomposition

Tensor term Separation rank Basis function Normalization constant

  • R. A. Harshman, J. D. Carroll, J.-J. Chang

14

slide-16
SLIDE 16

Low Rank Tensor Decomposition

Low rank tensor decomposition CANDECOMP/PARAFAC tensor decomposition

Function Operator

  • B. N. Khoromskij. Tensors-structured numerical methods in scientific computing: Survey
  • n recent advances. Chemometrics and Intelligent Laboratory Systems, Jan. 2012.
  • Y. Sun and M. Kumar. A tensor decomposition approach to high dimensional stationary

Fokker-Planck equations. ACC, 2014

15

slide-17
SLIDE 17

Linearly Solvable HJB Equation

16

Stochastic nonlinear affine system: Cost function:

slide-18
SLIDE 18

Linearly Solvable HJB Equation

Rewrite Tensor form

17

slide-19
SLIDE 19

Tensor Decomposition of HJB

18 x =

Boundary Solution Operator 1 1 1

slide-20
SLIDE 20

Linearly Solvable HJB Equation

Rewrite Tensor form Benefit: Memory and operations scale linearly with dimension

But, it is NP-Hard (J. Håstad)

19

slide-21
SLIDE 21

Outline

  • Motivation
  • Problem formulation
  • Low rank tensor decomposition
  • Alternating least squares & improvements
  • Example
  • Summary & future work

20

slide-22
SLIDE 22

Alternating Least Squares

1. Fix the separation rank, rF 2. For each dimension k, solve a least squares problem 3. Iterate through all k and repeat until the residual is small enough 4. Increase rF if residual cannot decrease anymore but it is still too large

21

regularization 𝛽 ~ 𝜈 approximately rounding error

  • G. Beylkin, M. J. Mohlenkamp
slide-23
SLIDE 23

Alternating Least Squares

22

Normal equation

  • G. Beylkin, M. J. Mohlenkamp

Complexity Accuracy Rounding error Number

  • f grid
slide-24
SLIDE 24

Issues with ALS

Ill-condition

Note:

  • A includes HJB and

boundary condition

  • F is the solution

23

slide-25
SLIDE 25

Sequential Computation of Solution

24 Residual Error

slide-26
SLIDE 26

Issues with ALS

Normal equation

Ill-condition

Note:

  • A includes HJB and

boundary condition

  • F is the solution

25

slide-27
SLIDE 27

Tensor Decomposition of HJB

26 x =

Boundary Solution Operator 1 1 1

slide-28
SLIDE 28

Rescaling Boundary Condition

27 Residual Error

slide-29
SLIDE 29

Sequential Alternating Least Squares

Modified ALS with:

  • Sequential computation of solution
  • Boundary condition rescaling

MATLAB code is available at http://www.cds.caltech.edu/~yleong/

28 Residual Error

slide-30
SLIDE 30

Sequential Alternating Least Squares

Modified ALS with:

  • Sequential computation of solution
  • Boundary condition rescaling

29

Recent work:

  • M. J. Reynolds, A. Doostan, G. Beylkin. Randomized Alternating Least Squares For

Canonical Tensor Decompositions: Application To A PDE With Random Data, SIAM J. Scientific Computing, 2016

Random matrix

slide-31
SLIDE 31

Outline

  • Motivation
  • Problem formulation
  • Low rank tensor decomposition
  • Alternating least squares & improvements
  • Example
  • Summary & future work

30

slide-32
SLIDE 32

Inverted Pendulum

31 Linux machine with 3 GHz i7 processor and 64 GB RAM MATLAB 2014a Solution rank, rF = 328

  • H. M. Osinga and J. Hauser, “The geometry of the solution set of nonlinear optimal control

problems,” J. Dynamics and Differential Equations, 2006.

Accuracy

Number of grid = 201

slide-33
SLIDE 33

Inverted Pendulum

32

slide-34
SLIDE 34

Vertical Take-off and Landing Aircraft

33

  • J. Hauser, S. Sastry, G. Meyer. Nonlinear

control design for slightly non-minimum phase systems: application to V/STOL

  • aircraft. Automatica. 1992

Number of grid = 100

slide-35
SLIDE 35

Vertical Take-off and Landing Aircraft

34 Number of grid = 100

slide-36
SLIDE 36

Quadcopter

35

  • L. R. Carrillo, A. E. López, R. Lozano,
  • C. Pégard. Modeling the quad-rotor

mini-rotorcraft. In Quad Rotorcraft Control 2013. Springer London.

Number of grid = 100

slide-37
SLIDE 37

Outline

  • Motivation
  • Problem formulation
  • Low rank tensor decomposition
  • Alternating least squares & improvements
  • Example
  • Summary & future work

36

slide-38
SLIDE 38

Summary

  • Low rank tensor decomposition allows for high

dimensional HJB representations and computations that scales linearly with dimensions

  • SeALS improves ALS by alleviating the ill-condition issue

(MATLAB code is available online at http://www.cds.caltech.edu/~yleong/) – Sequentially computing the solution – Rescaling boundary condition

37

slide-39
SLIDE 39

Future Work

  • Improve the algorithm using different discretization

schemes (e.g. Chebyshev spectral differentiation)

  • Analyze the algorithm more carefully to quantify

convergence and accuracy

  • Analyze the properties of the controller given by the

solution of SeALS

  • Apply to more difficult problems to find out when SeALS

breaks

38