Collaborators Joint work with Sarah Dean, Aurelia Guy, Horia Mania, - - PowerPoint PPT Presentation

Collaborators

Joint work with Sarah Dean, Aurelia Guy, Horia Mania, Nikolai Matni, Max Simchowitz, and Stephen Tu.

trustable, scalable, predictable

xt

u x e

Ct is the cost. If you maximize, it’s called a reward. ft is the state-transition function

is an observed trajectory τt = (u1, . . . , ut−1, x0, . . . , xt)

minimize Ee hPT

t=1 Ct(xt, ut)

i s.t. xt+1 = ft(xt, ut, et) ut = πt(τt)

is the policy. This is the optimization decision variable. πt(τt)

xt is the state, ut is the input, et is a noise process

Optimal control

Learning to control

xt

u x e

Ct is the cost. If you maximize, it’s called a reward. ft is the state-transition function

is an observed trajectory τt = (u1, . . . , ut−1, x0, . . . , xt)

minimize Ee hPT

t=1 Ct(xt, ut)

i s.t. xt+1 = ft(xt, ut, et) ut = πt(τt)

is the policy. This is the optimization decision variable. πt(τt)

Perennial challenge: how to perform optimal control when the system is unknown? unknown!

xt is the state, ut is the input, et is a noise process

How well must we understand a system in

• rder to control it?

∂ ∂t(ρu) + · (ρu u + pI) = · τ + ρg

M ˙ T = ˙ Q + ˙ mscp(Ts − T)

HVAC ROOM sensor state action

∂ ∂t(ρu) + · (ρu u + pI) = · τ + ρg

M ˙ T = ˙ Q + ˙ mscp(Ts − T)

HVAC ROOM sensor state action

• model predictive

control

• reinforcement

learning

• PID control?

Identify everything Identify a coarse model We don’t need no stinking models!

• High performance

aerodynamics

We need robust fundamentals to distinguish these approaches

But PID control works…

10

• 2

10

• 1

10 10

1

10

2

• 50
• 40
• 30
• 20
• 10

10 20 30 40 50 Magnitude (dB) Bode Diagram Frequency (rad/sec)

One decade Gain crossover point Loglog slope = -1.5 2 ≈ 6dB 0.5 ≈ -6dB

2 parameters suffice for 95% of all control applications. How much needs to be modeled for more advanced control? Can we learn to compensate for poor models, changing conditions?

Learning to control

xt

u x e

minimize Ee hPT

t=1 Ct(xt, ut)

i s.t. xt+1 = ft(xt, ut, et) ut = πt(τt)

What is the optimal estimation/design scheme? How many samples are needed for near optimal control?

Oracle: You can generate N trajectories of length T. Challenge: Build a controller with smallest error with fixed sampling budget (N x T).

RL Methods

G

xt

u x e

approximate dynamic programming model-based

• Model-based: fit model from data
• Model-free
• Direct policy search: search for actions from data
• Approximate dynamic programming
• estimate cost from data (with or without model)

direct policy search minimize Ee hPT

t=1 Ct(xt, ut)

i s.t. xt+1 = ft(xt, ut, et) ut = πt(τt)

How to solve optimal control when the model f is unknown?

Model-based RL

minimize Ee hPT

t=1 Ct(xt, ut)

i s.t. xt+1 = f(xt, ut, et) ut = πt(τt)

Collect some simulation data. Should have xt+1 ≈ ϕ(xt, ut) + νt

minimize Eω hPT

t=1 Ct(xt, ut)

i s.t. xt+1 = ϕ(xt, ut) + ωt ut = π(τt)

Solve approximate problem: Fit dynamics with supervised learning:

ˆ ϕ = arg min

ϕ T−1

X

t=0

||xt+1 − ϕ(xt, ut)||2

RL Methods

G

xt

u x e

approximate dynamic programming model-based direct policy search minimize Ee hPT

t=1 Ct(xt, ut)

i s.t. xt+1 = ft(xt, ut, et) ut = πt(τt)

• Model-based: fit model from data
• Model-free
• Direct policy search: search for actions from data
• Approximate dynamic programming
• estimate cost from data (with or without model)

How to solve optimal control when the model f is unknown?

“Simplest” Example: LQR

minimize E h

1 T

PT

t=1 x∗ t Qxt + u∗ t Rut

i s.t. xt+1 = Axt + But + et

• Optimization simplicity
• Elegant Dynamic Programming solutions
• Exact solution for baseline
• Static state feedback solution on infinite horizon
• Natural robustness
• Broadly applicable as is
• Core of many MPC and nonlinear control methods

“Simplest” Example: LQR

What is the optimal estimation/design scheme? How many samples are needed for near optimal control?

minimize E h

1 T

PT

t=1 x∗ t Qxt + u∗ t Rut

i s.t. xt+1 = Axt + But + et

Oracle: You can generate N trajectories of length T. Challenge: Build a controller with smallest error with fixed sampling budget (N x T).

Algorithm Samples per

• bservation

LQR parameters “optimal” error after NT steps Model-based d d2+dp ADP 1 Policy search 1 dp ✓d + p 2 ◆

ADP model-based policy search

x ∈ Rd

u ∈ Rp

Continuous Control:

Sample Complexity?

C C C r d + p NT

d + p √ NT

dp NT

minimize E hPT

t=1 x∗ t Qxt + u∗ t Rut

i s.t. xt+1 = Axt + But + et ut = Kxt

x ∈ Rd

u ∈ Rp

[Tu, R. 2018]

ADP vs model- based

minimize limT→∞ E h

1 T

PT

t=1 x∗ t Qxt + u∗ t Rut

i s.t. xt+1 = Axt + But + et ut = Kxt

Fix state feedback K, How to estimate value function? V(x) = x∗P?x

Use ADP (LSTD): ˆ Plstd

Estimate closed-loop model, solve DLE: ˆ Pplug

lim

T→∞ T E



• ˆ

Pplug − P?

• 2

F

• ≤ lim

T→∞ T E



• ˆ

Plstd − P?

• 2

F

• <latexit sha1_base64="6tr0GT8ImoUJZVK3f1GanCR3BY=">ADWHicrVJdb9MwFHVaYFv46sYjLxYV0pBgaiYkeEKT+Jb2UES7TapD5LhOas1xgn0DjbzwP+GVP4KTFIm2SLxwpcQn51zfm+vjuJDCwGj03ev1r12/sbO759+8dfvO3cH+wZnJS834lOUy1xcxNVwKxacgQPKLQnOaxZKfx5cvG/38C9dG5GoCVcHDjKZKJIJRcFQ0+EmkyCI7IVqkC6Ba518xESqBqsaTbySjsIhj+7rGRPIEZt1CrnyoGDHdWTbFJ3ZQpZpXeMneBwR4wr5XUVyFb35dIy7jxBj3xX43Lz/T1dpYP7PrtFgODoatYG3QbACQ7SKcbTvhWSeszLjCpikxsyCUQGhpRoEk7z2SWl4QdklTfnMQUzbkLbulHjh46Z4yTX7lGAW/bPHZmxlRZ7DKbKcym1pB/02YlJM9DK1RAlesa5SUEkOG2vxXGjOQFYOUKaF+1fMFlRTBu4CrHVpaxecrU1il6USLJ/zDVbCEjR1pOGQUeSm8q+FVLij1QZfNqc8W/VlW3kw1ciFWAen7pbph5tJTtDgs3j3wZnx0eBwx+eDk9erKzZRfRA3SIAvQMnaB3aIymiHnvdxbelXvRx/1d/p7XWrPW+25h9aif/ALe+YWZg=</latexit><latexit sha1_base64="6tr0GT8ImoUJZVK3f1GanCR3BY=">ADWHicrVJdb9MwFHVaYFv46sYjLxYV0pBgaiYkeEKT+Jb2UES7TapD5LhOas1xgn0DjbzwP+GVP4KTFIm2SLxwpcQn51zfm+vjuJDCwGj03ev1r12/sbO759+8dfvO3cH+wZnJS834lOUy1xcxNVwKxacgQPKLQnOaxZKfx5cvG/38C9dG5GoCVcHDjKZKJIJRcFQ0+EmkyCI7IVqkC6Ba518xESqBqsaTbySjsIhj+7rGRPIEZt1CrnyoGDHdWTbFJ3ZQpZpXeMneBwR4wr5XUVyFb35dIy7jxBj3xX43Lz/T1dpYP7PrtFgODoatYG3QbACQ7SKcbTvhWSeszLjCpikxsyCUQGhpRoEk7z2SWl4QdklTfnMQUzbkLbulHjh46Z4yTX7lGAW/bPHZmxlRZ7DKbKcym1pB/02YlJM9DK1RAlesa5SUEkOG2vxXGjOQFYOUKaF+1fMFlRTBu4CrHVpaxecrU1il6USLJ/zDVbCEjR1pOGQUeSm8q+FVLij1QZfNqc8W/VlW3kw1ciFWAen7pbph5tJTtDgs3j3wZnx0eBwx+eDk9erKzZRfRA3SIAvQMnaB3aIymiHnvdxbelXvRx/1d/p7XWrPW+25h9aif/ALe+YWZg=</latexit><latexit sha1_base64="6tr0GT8ImoUJZVK3f1GanCR3BY=">ADWHicrVJdb9MwFHVaYFv46sYjLxYV0pBgaiYkeEKT+Jb2UES7TapD5LhOas1xgn0DjbzwP+GVP4KTFIm2SLxwpcQn51zfm+vjuJDCwGj03ev1r12/sbO759+8dfvO3cH+wZnJS834lOUy1xcxNVwKxacgQPKLQnOaxZKfx5cvG/38C9dG5GoCVcHDjKZKJIJRcFQ0+EmkyCI7IVqkC6Ba518xESqBqsaTbySjsIhj+7rGRPIEZt1CrnyoGDHdWTbFJ3ZQpZpXeMneBwR4wr5XUVyFb35dIy7jxBj3xX43Lz/T1dpYP7PrtFgODoatYG3QbACQ7SKcbTvhWSeszLjCpikxsyCUQGhpRoEk7z2SWl4QdklTfnMQUzbkLbulHjh46Z4yTX7lGAW/bPHZmxlRZ7DKbKcym1pB/02YlJM9DK1RAlesa5SUEkOG2vxXGjOQFYOUKaF+1fMFlRTBu4CrHVpaxecrU1il6USLJ/zDVbCEjR1pOGQUeSm8q+FVLij1QZfNqc8W/VlW3kw1ciFWAen7pbph5tJTtDgs3j3wZnx0eBwx+eDk9erKzZRfRA3SIAvQMnaB3aIymiHnvdxbelXvRx/1d/p7XWrPW+25h9aif/ALe+YWZg=</latexit><latexit sha1_base64="6tr0GT8ImoUJZVK3f1GanCR3BY=">ADWHicrVJdb9MwFHVaYFv46sYjLxYV0pBgaiYkeEKT+Jb2UES7TapD5LhOas1xgn0DjbzwP+GVP4KTFIm2SLxwpcQn51zfm+vjuJDCwGj03ev1r12/sbO759+8dfvO3cH+wZnJS834lOUy1xcxNVwKxacgQPKLQnOaxZKfx5cvG/38C9dG5GoCVcHDjKZKJIJRcFQ0+EmkyCI7IVqkC6Ba518xESqBqsaTbySjsIhj+7rGRPIEZt1CrnyoGDHdWTbFJ3ZQpZpXeMneBwR4wr5XUVyFb35dIy7jxBj3xX43Lz/T1dpYP7PrtFgODoatYG3QbACQ7SKcbTvhWSeszLjCpikxsyCUQGhpRoEk7z2SWl4QdklTfnMQUzbkLbulHjh46Z4yTX7lGAW/bPHZmxlRZ7DKbKcym1pB/02YlJM9DK1RAlesa5SUEkOG2vxXGjOQFYOUKaF+1fMFlRTBu4CrHVpaxecrU1il6USLJ/zDVbCEjR1pOGQUeSm8q+FVLij1QZfNqc8W/VlW3kw1ciFWAen7pbph5tJTtDgs3j3wZnx0eBwx+eDk9erKzZRfRA3SIAvQMnaB3aIymiHnvdxbelXvRx/1d/p7XWrPW+25h9aif/ALe+YWZg=</latexit>

Thm:

lim

T→∞ T E



• ˆ

Pplug − P?

• 2

F

• ≤ O(d2)

There exist instances where and no algorithm can do better than O(d2).

lim

T→∞ T E



• ˆ

Plstd − P?

• 2

F

• ≥ Ω(d3)

Direct Policy Search vs model-based

x ∈ Rd

u ∈ Rp

[Tu, R. 2018]

minimize E hPT

t=1 x∗ t Qxt + u∗ t Rut

i s.t. xt+1 = Axt + But + et ut = Kxt

Find state feedback K. How does its cost compare to optimal J⋆? There exist instances where d>p and and no algorithm can do better than O(p2/d).

(with a sophisticated uncomputable baseline) lim inf

N→∞ N E

h J(ˆ Kpg)/J? − 1 i ≥ Ω(T2(p4/d + d2p))

lim

N→∞ N E

h J(ˆ Kplug)/J? − 1 i = O(p)

lim inf

N→∞ N E

h J(ˆ Kpg)/J? − 1 i ≥ Ω(p3/d + p2)

Estimate state-space model, solve DARE. Run policy gradient.

“Simplest” Example: LQR

+ru2

t

subject to xt+1 =

 1 1 1

• xt +

 1/m

• ut

xt =  zt vt

• <latexit sha1_base64="+ojsm2yurvuosb1Y5f0dyh5w9EA=">ACo3icbVHbtNAEN2YWym3FB5WRGBioSCjZBaHkCVQIKHPBRoaKXYisbriTPqem3tjqsEK5/B1/AKH8HfsE4DIgkjrfbonLlPWmlyHIa/OsGVq9eu39i5uXvr9p2797p797+4srYKh6rUpT1LwaEmg0Mm1nhWYQi1Xianr9t9dMLtI5Kc8LzCpMCckMTUsCeGnefz8YsX8s4xZxMkxbAlmYL+dWzcSwv/CdjNlfZdzthf1waXIbRCvQEys7Hu91kjgrV2gYaXBuVEUVpw0YJmUxsVuXDusQJ1DjiMPDRTokmY52UI+9kwmJ6X1z7Bcsv9GNFA4Ny9S7+kbnLpNrSX/p41qnhwmDZmqZjTqstCk1pJL2a5JZmRsZ57AMqS71WqKVhQ7Je5VmWZu0K1Nkzqw2pMsMNVvOMLXjSIRdAp2qeU9ay89gnBxQPuU/qk/byvKCd2zwb+YubplrM/SLS5/m0wfNF/1Y8+vuwdvVldZkc8FI/EvojEgTgSH8SxGAolvonv4of4GTwJBsGn4OTSNeisYh6INQuS31Rr01Q=</latexit><latexit sha1_base64="+ojsm2yurvuosb1Y5f0dyh5w9EA=">ACo3icbVHbtNAEN2YWym3FB5WRGBioSCjZBaHkCVQIKHPBRoaKXYisbriTPqem3tjqsEK5/B1/AKH8HfsE4DIgkjrfbonLlPWmlyHIa/OsGVq9eu39i5uXvr9p2797p797+4srYKh6rUpT1LwaEmg0Mm1nhWYQi1Xianr9t9dMLtI5Kc8LzCpMCckMTUsCeGnefz8YsX8s4xZxMkxbAlmYL+dWzcSwv/CdjNlfZdzthf1waXIbRCvQEys7Hu91kjgrV2gYaXBuVEUVpw0YJmUxsVuXDusQJ1DjiMPDRTokmY52UI+9kwmJ6X1z7Bcsv9GNFA4Ny9S7+kbnLpNrSX/p41qnhwmDZmqZjTqstCk1pJL2a5JZmRsZ57AMqS71WqKVhQ7Je5VmWZu0K1Nkzqw2pMsMNVvOMLXjSIRdAp2qeU9ay89gnBxQPuU/qk/byvKCd2zwb+YubplrM/SLS5/m0wfNF/1Y8+vuwdvVldZkc8FI/EvojEgTgSH8SxGAolvonv4of4GTwJBsGn4OTSNeisYh6INQuS31Rr01Q=</latexit><latexit sha1_base64="+ojsm2yurvuosb1Y5f0dyh5w9EA=">ACo3icbVHbtNAEN2YWym3FB5WRGBioSCjZBaHkCVQIKHPBRoaKXYisbriTPqem3tjqsEK5/B1/AKH8HfsE4DIgkjrfbonLlPWmlyHIa/OsGVq9eu39i5uXvr9p2797p797+4srYKh6rUpT1LwaEmg0Mm1nhWYQi1Xianr9t9dMLtI5Kc8LzCpMCckMTUsCeGnefz8YsX8s4xZxMkxbAlmYL+dWzcSwv/CdjNlfZdzthf1waXIbRCvQEys7Hu91kjgrV2gYaXBuVEUVpw0YJmUxsVuXDusQJ1DjiMPDRTokmY52UI+9kwmJ6X1z7Bcsv9GNFA4Ny9S7+kbnLpNrSX/p41qnhwmDZmqZjTqstCk1pJL2a5JZmRsZ57AMqS71WqKVhQ7Je5VmWZu0K1Nkzqw2pMsMNVvOMLXjSIRdAp2qeU9ay89gnBxQPuU/qk/byvKCd2zwb+YubplrM/SLS5/m0wfNF/1Y8+vuwdvVldZkc8FI/EvojEgTgSH8SxGAolvonv4of4GTwJBsGn4OTSNeisYh6INQuS31Rr01Q=</latexit><latexit sha1_base64="+ojsm2yurvuosb1Y5f0dyh5w9EA=">ACo3icbVHbtNAEN2YWym3FB5WRGBioSCjZBaHkCVQIKHPBRoaKXYisbriTPqem3tjqsEK5/B1/AKH8HfsE4DIgkjrfbonLlPWmlyHIa/OsGVq9eu39i5uXvr9p2797p797+4srYKh6rUpT1LwaEmg0Mm1nhWYQi1Xianr9t9dMLtI5Kc8LzCpMCckMTUsCeGnefz8YsX8s4xZxMkxbAlmYL+dWzcSwv/CdjNlfZdzthf1waXIbRCvQEys7Hu91kjgrV2gYaXBuVEUVpw0YJmUxsVuXDusQJ1DjiMPDRTokmY52UI+9kwmJ6X1z7Bcsv9GNFA4Ny9S7+kbnLpNrSX/p41qnhwmDZmqZjTqstCk1pJL2a5JZmRsZ57AMqS71WqKVhQ7Je5VmWZu0K1Nkzqw2pMsMNVvOMLXjSIRdAp2qeU9ay89gnBxQPuU/qk/byvKCd2zwb+YubplrM/SLS5/m0wfNF/1Y8+vuwdvVldZkc8FI/EvojEgTgSH8SxGAolvonv4of4GTwJBsGn4OTSNeisYh6INQuS31Rr01Q=</latexit>

minimize

T

X

t=0

(xt)2

1

samples

nominal control with 10 samples

argmin.net/code/lqr_policy_comparisons.ipynb

Extraordinary Claims Require Extraordinary Evidence*

“How can we dismiss an entire field which claims such success?” * only if your prior is correct

“Reinforcement learning results are tricky to reproduce: performance is very noisy, algorithms have many moving parts which allow for subtle bugs, and many papers don’t report all the required tricks.” “RL algorithms are challenging to implement correctly; good results typically only come after fixing many seemingly-trivial bugs.”

0.00 0.25 0.50 0.75 1.00 1.25 1.50 1.75 2.00

Timesteps

1000 2000 3000 4000 5000

Average Return HalfCheetah-v1 (TRPO, Different Random Seeds)

Random Average (5 runs) Random Average (5 runs)

0.00 0.25 0.50 0.75 1.00 1.25 1.50 1.75 2.00

Timesteps

−500 500 1000 1500 2000

Average Return HalfCheetah-v1 (TRPO, Codebase Comparison)

blog.openai.com/openai-baselines-dqn/

There has to be a better way!

arxiv:1709.06560

G K

u y

Coarse-ID control

^

Coarse-grained model is trivial to fit

Δ

w v High dimensional stats bounds the error Design robust control for feedback loop Robust certainty equivalence.

Coarse-ID Control for LQR

[Dean, Mania, Matni, R.,Tu, 2017] Gaussian noise

Run an experiment for T steps with random input. Then minimize(A,B) PT

i=1 kxi+1 Axi Buik2

minimize limT→∞ E h

1 T

PT

t=1 x∗ t Qxt + u∗ t Rut

i s.t. xt+1 = Axt + But + et

[Mania, Jordan, R., Simchowitz, Tu, 2018]

How many samples are needed to Estimate (A,B)? (A stable) If A ˆ A B ˆ B and then T ≥ ˜ O ✓2(d + p) ✏2 ◆

w.h.p.

Coarse-ID Control for LQR

[Dean, Mania, Matni, R., Tu 2017]

minimize

u

sup

k∆Ak2✏A, k∆Bk2✏B

lim

T!1 1 T

PT

t=1 x⇤ t Qxt + u⇤ t Rut

s.t. xt+1 = (ˆ A + ∆A)xt + (ˆ B + ∆B)ut Robust LQR solution via robust system level synthesis Solving an SDP relaxation of this robust control problem yields w.h.p.

J(ˆ K) − J? J? ≤ C r σ2(d + p) T

J(ˆ K)

= LQR cost of coarse-ID control J?

= optimal LQR when (A,B) known

This also tells you when your cost is finite!

Why robust?

Slightly unstable system, system ID tends to think some nodes are stable

xt+1 =   1.01 0.01 0.01 1.01 0.01 0.01 1.01   xt +   1 1 1   ut + et

Least-squares estimate may yield unstable controller Robust synthesis yields stable controller

Model-free performs worse than model-based

Even LQR is not simple!!!

Gaussian noise

minimize J := ∞

t=1 xT t Qxt + uT t Rut

s.t. xt+1 = Axt + But + et

• Coarse-ID control is the first non-asymptotic bound for LQR in

this oracle model.

• The estimation results disagree with 50 papers on Cosma

Shalizi’s blog.

• The guarantees for least-squares estimation required some heavy

machinery and are building on results from last year.

• The SDP relaxation uses brand new techniques in controller

parameterization (Systems Level Synthesis by Matni et al.)

• Key insight: Robustness makes analysis tractable

Even the simplest RL problems are really hard.

• Poly-time Direct Adaptive Control (NeurIPS2018)
• Limitations of policy search (NeurIPS 2018)
• Safe exploration (submitted to ACC 2019)

Extensions So far…

• Model based methods seem to perform better than

model free ones in theory and practice.

• RL needs better baselines!
• Simple algorithms seem to be surprisingly competitive.
• Analysis of time series is annoyingly hard!

The Linearization Principle

If a machine learning algorithm does crazy things when restricted to linear models, it’s going to do crazy things on complex nonlinear models too. What happens when we return to nonlinear models?

Simple, static, linear policies outperform deep RL and can be found with simple, standard

• ptimization algorithms.

Model Predictive Control

• utperforms any direct

policy search method. [Mania, Guy, R. 2018] Video from Todorov Lab, 2012

What is ML good for in control?

• Fundamentally, almost all machine learning successes are

in nonparametric prediction (mostly classification).

Perceptual sensors in the loop Forecasting in MPC

How to incorporate uncertain predictive perception in trustable, scalable, predictable autonomy?

Collaborators

Joint work with Sarah Dean, Aurelia Guy, Horia Mania, Nikolai Matni, Max Simchowitz, and Stephen Tu.

References

• “On the Sample Complexity of the Linear Quadratic Regulator.” S. Dean, H. Mania, N. Matni,
• B. Recht, and S. Tu. arXiv:1710.01688
• “Least-squares Temporal Differencing for the Linear Quadratic Regulator” S. Tu and B. Recht.

In ICML 2018. arXiv:1712.08642

• “Learning without Mixing.” H. Mania, M. I. Jordan, B. Recht, M. Simchowitz, and S. Tu. In COLT
• 2018. arXiv:1802.08334
• “Simple random search provides a competitive approach to reinforcement learning.” H.

Mania, A. Guy, and B. Recht. In NeurIPS 2018. arXiv:1803.07055

• “Regret Bounds for Robust Adaptive Control of the Linear Quadratic Regulator.” S. Dean, H.

Mania, N. Matni, B. Recht, and S. Tu. In NeurIPS 2018. arXiv:1805.09388

• “The Gap Between Model-Based and Model-Free Methods on the Linear Quadratic

Regulator: An Asymptotic Viewpoint.” S. Tu and B. Recht. arXiv:1812.03565

• A Tour of Reinforcement Learning: The View from Continuous Control.” B. Recht.

arXiv:1806.09460