Solving High Dimensional Hamilton- Jacobi-Bellman Equations Using - PowerPoint PPT Presentation

Solving High Dimensional Hamilton- Jacobi-Bellman Equations Using Low Rank Tensor Decomposition Yoke Peng Leong California Institute of Technology Joint work with Elis Stefansson, Matanya Horowitz, Joel Burdick

1 Motivation Synthesize optimal feedback controllers for nonlinear dynamical systems in high dimensions > 40 degree of freedoms > 12 degree of freedoms

2 Outline • Motivation • Problem formulation • Low rank tensor decomposition • Alternating least squares & improvements • Example • Summary & future work

3 Problem Formulation Stochastic nonlinear affine system: Brownian noise Synthesize a controller, u(x), that minimize the cost function:

4 HJB Equation Value function (cost-to-go): Dynamic programming gives the HJB equation Nonlinear PDE where the optimal controller is given by

5 Linearly Solvable HJB Equation HJB equation (a nonlinear PDE) Condition: Any system of the following form can satisfy the condition Desirability Log transformation: function W. H. Fleming, C. J. Holland, P. Dai Pra, R. Filliger, H. Kappen, E. Todorov, F. Stulp, E. A. Theodorou, K. Dvijotham, S. Schaal Linearly solvable HJB equation (a linear PDE)

6 Convex Optimization Relaxation Sum of squares program Upper bound solution Suboptimal Stabilizing Controllers for Linearly Solvable Systems, Y. P. Leong, M. B. Horowitz, J. W. Burdick, CDC 2015 Linearly Solvable Stochastic Control Lyapunov Functions, Y. P. Leong, M. B. Horowitz, J. W. Burdick, SIAM Journal on Control and Optimization, Accepted

7 Main Results Theorem: The approximate solution is a stochastic control Lyapunov function (SCLF). Proof: Relaxed HJB and satisfies the definition of SCLF. Corollary: The suboptimal controller is stabilizing in probability. The approximate solution gives a suboptimal stabilizing controller. Theorem: Given the controller , then Expected cost of a system using the given controller Proof: Manipulate the relaxed HJB and the error bound of approximate value function. The approximate solution gives an upper bound to the actual cost when using the suboptimal controller.

8 Convex Optimization Relaxation Sum of squares program Problem: Curse of dimensionality Upper bound solution Suboptimal Stabilizing Controllers for Linearly Solvable Systems, Y. P. Leong, M. B. Horowitz, J. W. Burdick, CDC 2015 Linearly Solvable Stochastic Control Lyapunov Functions, Y. P. Leong, M. B. Horowitz, J. W. Burdick, SIAM Journal on Control and Optimization, Accepted

9 Related Works • Sparse grid approximation (J. Garcke, A. Kröner) • Taylor polynomial approximation + patchy technique (C. O. Aguilar, A. J. Krener) • Max-plus expansion (W. M. McEneaney) • Model reduction (K. Kunisch, S. Volkwein, L. Xie, S. Gombao) • Level-set algorithm (I. M. Mitchell, C. J. Tomlin) etc…

10 Goal Solve linear HJB equations for high dimensional dynamical systems > 40 degree of freedoms > 12 degree of freedoms

11 Outline • Motivation • Problem formulation • Low rank tensor decomposition • Alternating least squares & improvements • Example • Summary & future work

12 Low Rank Tensor Decomposition G. Beylkin Separated representation M. J. Mohlenkamp Low rank tensor decomposition

13 Low Rank Tensor Decomposition Low rank tensor decomposition

14 Low Rank Tensor Decomposition Low rank tensor decomposition CANDECOMP/PARAFAC tensor decomposition R. A. Harshman, J. D. Carroll, J.-J. Chang Separation rank Basis function Tensor term Normalization constant

15 Low Rank Tensor Decomposition Low rank tensor decomposition CANDECOMP/PARAFAC tensor decomposition Operator Function B. N. Khoromskij. Tensors-structured numerical methods in scientific computing: Survey on recent advances. Chemometrics and Intelligent Laboratory Systems, Jan. 2012. Y. Sun and M. Kumar. A tensor decomposition approach to high dimensional stationary Fokker-Planck equations. ACC, 2014

16 Linearly Solvable HJB Equation Stochastic nonlinear affine system: Cost function:

17 Linearly Solvable HJB Equation Rewrite Tensor form

18 Tensor Decomposition of HJB Boundary 1 x 1 = 1 Solution Operator

19 Linearly Solvable HJB Equation Rewrite Tensor form Benefit: Memory and operations scale linearly with dimension But, it is NP-Hard (J. Håstad)

20 Outline • Motivation • Problem formulation • Low rank tensor decomposition • Alternating least squares & improvements • Example • Summary & future work

21 Alternating Least Squares G. Beylkin, M. J. Mohlenkamp regularization 𝛽 ~ 𝜈 approximately rounding error 1. Fix the separation rank, r F 2. For each dimension k, solve a least squares problem 3. Iterate through all k and repeat until the residual is small enough 4. Increase r F if residual cannot decrease anymore but it is still too large

22 Alternating Least Squares G. Beylkin, M. J. Mohlenkamp Normal equation Complexity Number Accuracy of grid Rounding error

23 Issues with ALS Ill-condition Note: • A includes HJB and boundary condition • F is the solution

24 Sequential Computation of Solution Residual Error

25 Issues with ALS Normal equation Ill-condition Note: • A includes HJB and boundary condition • F is the solution

26 Tensor Decomposition of HJB Boundary 1 x 1 = 1 Solution Operator

27 Rescaling Boundary Condition Residual Error

28 Sequential Alternating Least Squares Modified ALS with: • Sequential computation of solution • Boundary condition rescaling Residual Error MATLAB code is available at http://www.cds.caltech.edu/~yleong/

29 Sequential Alternating Least Squares Modified ALS with: • Sequential computation of solution • Boundary condition rescaling Recent work: M. J. Reynolds, A. Doostan, G. Beylkin. Randomized Alternating Least Squares For Canonical Tensor Decompositions: Application To A PDE With Random Data, SIAM J. Scientific Computing, 2016 Random matrix

30 Outline • Motivation • Problem formulation • Low rank tensor decomposition • Alternating least squares & improvements • Example • Summary & future work

31 Inverted Pendulum Linux machine with 3 GHz i7 processor and 64 GB RAM MATLAB 2014a Number of Solution rank, r F = 328 grid = 201 Accuracy H. M. Osinga and J. Hauser, “The geometry of the solution set of nonlinear optimal control problems,” J. Dynamics and Differential Equations , 2006.

32 Inverted Pendulum

33 Vertical Take-off and Landing Aircraft Number of grid = 100 J. Hauser, S. Sastry, G. Meyer. Nonlinear control design for slightly non-minimum phase systems: application to V/STOL aircraft. Automatica. 1992

34 Vertical Take-off and Landing Aircraft Number of grid = 100

35 Quadcopter Number of grid = 100 L. R. Carrillo, A. E. López, R. Lozano, C. Pégard. Modeling the quad-rotor mini-rotorcraft. In Quad Rotorcraft Control 2013. Springer London.

36 Outline • Motivation • Problem formulation • Low rank tensor decomposition • Alternating least squares & improvements • Example • Summary & future work

37 Summary • Low rank tensor decomposition allows for high dimensional HJB representations and computations that scales linearly with dimensions • SeALS improves ALS by alleviating the ill-condition issue (MATLAB code is available online at http://www.cds.caltech.edu/~yleong/) – Sequentially computing the solution – Rescaling boundary condition

38 Future Work • Improve the algorithm using different discretization schemes (e.g. Chebyshev spectral differentiation) • Analyze the algorithm more carefully to quantify convergence and accuracy • Analyze the properties of the controller given by the solution of SeALS • Apply to more difficult problems to find out when SeALS breaks