learning to control the linear quadratic regulator Benjamin Recht - - PowerPoint PPT Presentation

learning to control the linear quadratic regulator

Benjamin Recht University of California, Berkeley

Collaborators

Joint work with Sarah Dean, Horia Mania, Nikolai Matni, Max Simchowitz, and Stephen Tu.

trustable, scalable, predictable

What are the fundamental limits of learning systems that interact with the physical environment?

theoretical foundations

• statistical learning theory
• robust control theory
• core optimization

How well must we understand a system in order to control it?

Optimal control

xt

u x e

Ct is the cost. If you maximize, it’s called a reward. xt is the state, ut is the input, et is a noise process ft is the state-transition function

is an observed trajectory τt = (u1, . . . , ut−1, x0, . . . , xt)

minimize Ee hPT

t=1 Ct(xt, ut)

i s.t. xt+1 = ft(xt, ut, et) ut = πt(τt)

is the policy. This is the optimization decision variable. πt(τt)

Learning to control

xt

u x e

Ct is the cost. If you maximize, it’s called a reward. ft is the state-transition function

is an observed trajectory τt = (u1, . . . , ut−1, x0, . . . , xt)

minimize Ee hPT

t=1 Ct(xt, ut)

i s.t. xt+1 = ft(xt, ut, et) ut = πt(τt)

is the policy. This is the optimization decision variable. πt(τt)

Perennial challenge: how to perform optimal control when the system is unknown? unknown!

xt is the state, ut is the input, et is a noise process

RL Triopoly

G

xt

u x e

approximate dynamic programming model-based

• Model-based: fit model from data
• Model-free
• Approximate dynamic programming: estimate cost from data
• Direct policy search: search for actions from data

direct policy search minimize Ee hPT

t=1 Ct(xt, ut)

i s.t. xt+1 = ft(xt, ut, et) ut = πt(τt)

How to solve optimal control when the model f is unknown?

Model-based RL

minimize Ee hPT

t=1 Ct(xt, ut)

i s.t. xt+1 = f(xt, ut, et) ut = πt(τt)

Collect some simulation data. Should have xt+1 ≈ ϕ(xt, ut) + νt

minimize Eω hPT

t=1 Ct(xt, ut)

i s.t. xt+1 = ϕ(xt, ut) + ωt ut = π(τt)

Solve approximate problem: Fit dynamics with supervised learning:

ˆ ϕ = arg min

ϕ T−1

X

t=0

||xt+1 − ϕ(xt, ut)||2

“Simplest” Example: LQR

minimize E h

1 T

PT

t=1 x∗ t Qxt + u∗ t Rut

i s.t. xt+1 = Axt + But + et

• Optimization simplicity
• Elegant Dynamic Programming solutions
• Exact solution for baseline
• Natural robustness
• Broadly applicable as is
• Core of many MPC and nonlinear control methods
• Useful model for sensorimotor modeling

“Simplest” Example: LQR

What is the optimal estimation/design scheme? How many samples are needed for near optimal control?

minimize E h

1 T

PT

t=1 x∗ t Qxt + u∗ t Rut

i s.t. xt+1 = Axt + But + et

Oracle: You can generate N trajectories of length T. Challenge: Build a controller with smallest error with fixed sampling budget (N x T).

Model-based LQR

minimize E h

1 T

PT

t=1 x∗ t Qxt + u∗ t Rut

i s.t. xt+1 = Axt + But + et

Fit dynamics with supervised learning: minimize(A,B) PT

i=1 kxi+1 Axi Buik2

Solve approximate problem: minimize E h

1 T

PT

t=1 x∗ t Qxt + u∗ t Rut

i s.t. xt+1 = ˆ Axt + ˆ But + ωt

Collect some simulation data. Will have xt+1 = Axt + But + et

G K

u y

Coarse-ID control

^

Coarse-grained model is trivial to fit

Δ

w v High dimensional stats bounds the error Design robust control for feedback loop Robust certainty equivalence.

“Simple” Example: LQR

How many samples are needed to Estimate (A,B)? (A stable)

[Dean, Mania, Matni, R.,Tu, 2017] Gaussian noise

Run an experiment for T steps with random input. Then minimize(A,B) PT

i=1 kxi+1 Axi Buik2

[Mania, Jordan, R., Simchowitz, Tu, 2018]

controllability Gramian where Λc = AΛcA∗ + BB∗ If T ≥ ˜ O ✓ 2(d + p) min(Λc)✏2 ◆

A ˆ A B ˆ B and then w.h.p. minimize limT→∞ E h

1 T

PT

t=1 x∗ t Qxt + u∗ t Rut

i s.t. xt+1 = Axt + But + et

Similar result for non-stable A.

x ∈ Rd

u ∈ Rp

“Simple” Example: LQR

[Dean, Mania, Matni, R., Tu 2017]

“Obvious strategy”: Estimate , build control ut = ˆ Kxt (ˆ A, ˆ B) Λc = AΛcA∗ + BB∗

controllability Gramian closed loop gain Γcl := k(zI A BK?)−1kH∞

minimize

u

sup

k∆Ak2✏A, k∆Bk2✏B

lim

T!1 1 T

PT

t=1 x⇤ t Qxt + u⇤ t Rut

s.t. xt+1 = (ˆ A + ∆A)xt + (ˆ B + ∆B)ut This also tells you when your cost is finite! Solving an SDP relaxation of this robust control problem yields

J(ˆ K) J? J?  C Γcl ⇣ λmin(Λc)−1/2 + kK?k2 ⌘ r σ2(d + p) T

w.h.p.

minimize

u

lim

T→∞ E

h

1 T

PT

t=1 x∗ t Qxt + u∗ t Rut

i s.t. xt+1 = Axt + But + et ut =

t

X

k=1

Φu[k]et−k

Key to formulation: Write u as LTI function of

• disturbance. (Disturbance feedback)

xt

u x e

minimize

u

lim

T→∞ E

h

1 T

PT

t=1 x∗ t Qxt + u∗ t Rut

i s.t. xt+1 = Axt + But + et

xt

u x e

 xt ut

• =

t

X

k=1

Φx[k] Φu[k]

• et−k

Key to formulation: Write u as LTI function of

• disturbance. (Disturbance feedback)

Then x is a linear function of the disturbance as well. In closed loop, can’t decouple these boxes: consider the mapping from disturbance to both signals.

K = ΦuΦ−1

x

K

u

x

x e u Φu

x e Φ−1

x

E [x∗

t Qxt] = σ2 t

X

k=1

Tr(Φx[k]∗QΦx[k]) E [u∗

t Rut] = σ2 t

X

k=1

Tr(Φu[k]∗RΦu[k])

 xt ut

• =

t

X

k=1

Φx[k] Φu[k]

• et−k

Key to formulation: Write u as LTI function of

• disturbance. (Disturbance feedback)

Then x is a linear function of the disturbance as well.

⇥ zI − A −B⇤ Φx Φu

• = I

zΦxe = AΦxe + BΦue + e

Dynamic equality constraint implies:

minimize

u

lim

T→∞ E

h

1 T

PT

t=1 x∗ t Qxt + u∗ t Rut

i s.t. xt+1 = Axt + But + et

xt

u x e

 xt ut

• =

t

X

k=1

Φx[k] Φu[k]

• et−k

Key to formulation: Write u as LTI function of

• disturbance. (Disturbance feedback)

Then x is a linear function of the disturbance as well.

System Level Synthesis

minimize

Φ

• "

Q

1 2

R

1 2

# Φx Φu

• 2

H2

s.t. ⇥ zI − A −B⇤  Φx Φu

• = I

minimize

u

lim

T→∞ E

h

1 T

PT

t=1 x∗ t Qxt + u∗ t Rut

i s.t. xt+1 = Axt + But + et

xt

u x e

Suppose (i.e., nominal + error). Note that if

⇥ zI − A −B⇤ Φx Φu

• = I +

⇥∆A ∆B ⇤ Φx Φu

• <latexit sha1_base64="vhGwz9WNMmJoQWmFYd6EJI1a2P8=">ADOHicpVJNb9NAEF2brxK+UnrksiICtYJWNkKC1JbKtFKlQiCtEGxZa3XE2fV9draHaMEK3+MK7+CGzfUK7+AdWIEcbgx0kpP73Z2ZnZuJDCoOd9c9wrV69dv7Fxs3Pr9p2797qb989MXmoOA57LXA9jZkAKBQMUKGFYaGBZLOE8vnhd6+efQBuRqw84KyDMWKrEWHCGloq6X2kQypUFWcMtZjO6eT3QP6mO4e0gBU8odv+4L+RERTGgRLVLbsr+gJ7TxZzoCiSyqKzTw/+p0bETdnrfnLYKuA78BPdJEP9p0wiDJeZmBQi6ZMSPfKzCsmEbBJcw7QWmgYPyCpTCyULEMTFgtpj2njyT0HGu7VFIF+zfGRXLjJlsXaR05MW6vJf2mjEscvw0qokRQfFloXEqKOa1XRxOhgaOcWcC4FvatlE+YZhztgleqLO4ugK90Uk1LJXieQIuVOEXNLGkAMyZU3VX1RkhJ3zNl6KlIJ/hbtdfW8vaRSAWap6f2F6mdNbNdiN8e/zo4e7bnW/zueW/sFnNBnlAHpJt4pMXZJ8ckz4ZEO7sOG+dofPR/eJ+d3+4l0ur6zQ5W2Ql3J+/AN6PCVU=</latexit><latexit sha1_base64="vhGwz9WNMmJoQWmFYd6EJI1a2P8=">ADOHicpVJNb9NAEF2brxK+UnrksiICtYJWNkKC1JbKtFKlQiCtEGxZa3XE2fV9draHaMEK3+MK7+CGzfUK7+AdWIEcbgx0kpP73Z2ZnZuJDCoOd9c9wrV69dv7Fxs3Pr9p2797qb989MXmoOA57LXA9jZkAKBQMUKGFYaGBZLOE8vnhd6+efQBuRqw84KyDMWKrEWHCGloq6X2kQypUFWcMtZjO6eT3QP6mO4e0gBU8odv+4L+RERTGgRLVLbsr+gJ7TxZzoCiSyqKzTw/+p0bETdnrfnLYKuA78BPdJEP9p0wiDJeZmBQi6ZMSPfKzCsmEbBJcw7QWmgYPyCpTCyULEMTFgtpj2njyT0HGu7VFIF+zfGRXLjJlsXaR05MW6vJf2mjEscvw0qokRQfFloXEqKOa1XRxOhgaOcWcC4FvatlE+YZhztgleqLO4ugK90Uk1LJXieQIuVOEXNLGkAMyZU3VX1RkhJ3zNl6KlIJ/hbtdfW8vaRSAWap6f2F6mdNbNdiN8e/zo4e7bnW/zueW/sFnNBnlAHpJt4pMXZJ8ckz4ZEO7sOG+dofPR/eJ+d3+4l0ur6zQ5W2Ql3J+/AN6PCVU=</latexit><latexit sha1_base64="vhGwz9WNMmJoQWmFYd6EJI1a2P8=">ADOHicpVJNb9NAEF2brxK+UnrksiICtYJWNkKC1JbKtFKlQiCtEGxZa3XE2fV9draHaMEK3+MK7+CGzfUK7+AdWIEcbgx0kpP73Z2ZnZuJDCoOd9c9wrV69dv7Fxs3Pr9p2797qb989MXmoOA57LXA9jZkAKBQMUKGFYaGBZLOE8vnhd6+efQBuRqw84KyDMWKrEWHCGloq6X2kQypUFWcMtZjO6eT3QP6mO4e0gBU8odv+4L+RERTGgRLVLbsr+gJ7TxZzoCiSyqKzTw/+p0bETdnrfnLYKuA78BPdJEP9p0wiDJeZmBQi6ZMSPfKzCsmEbBJcw7QWmgYPyCpTCyULEMTFgtpj2njyT0HGu7VFIF+zfGRXLjJlsXaR05MW6vJf2mjEscvw0qokRQfFloXEqKOa1XRxOhgaOcWcC4FvatlE+YZhztgleqLO4ugK90Uk1LJXieQIuVOEXNLGkAMyZU3VX1RkhJ3zNl6KlIJ/hbtdfW8vaRSAWap6f2F6mdNbNdiN8e/zo4e7bnW/zueW/sFnNBnlAHpJt4pMXZJ8ckz4ZEO7sOG+dofPR/eJ+d3+4l0ur6zQ5W2Ql3J+/AN6PCVU=</latexit><latexit sha1_base64="vhGwz9WNMmJoQWmFYd6EJI1a2P8=">ADOHicpVJNb9NAEF2brxK+UnrksiICtYJWNkKC1JbKtFKlQiCtEGxZa3XE2fV9draHaMEK3+MK7+CGzfUK7+AdWIEcbgx0kpP73Z2ZnZuJDCoOd9c9wrV69dv7Fxs3Pr9p2797qb989MXmoOA57LXA9jZkAKBQMUKGFYaGBZLOE8vnhd6+efQBuRqw84KyDMWKrEWHCGloq6X2kQypUFWcMtZjO6eT3QP6mO4e0gBU8odv+4L+RERTGgRLVLbsr+gJ7TxZzoCiSyqKzTw/+p0bETdnrfnLYKuA78BPdJEP9p0wiDJeZmBQi6ZMSPfKzCsmEbBJcw7QWmgYPyCpTCyULEMTFgtpj2njyT0HGu7VFIF+zfGRXLjJlsXaR05MW6vJf2mjEscvw0qokRQfFloXEqKOa1XRxOhgaOcWcC4FvatlE+YZhztgleqLO4ugK90Uk1LJXieQIuVOEXNLGkAMyZU3VX1RkhJ3zNl6KlIJ/hbtdfW8vaRSAWap6f2F6mdNbNdiN8e/zo4e7bnW/zueW/sFnNBnlAHpJt4pMXZJ8ckz4ZEO7sOG+dofPR/eJ+d3+4l0ur6zQ5W2Ql3J+/AN6PCVU=</latexit>

=: I + ∆

(A, B) = (ˆ A + ∆A, ˆ B + ∆B)

⇥ zI − A −B⇤ Φx Φu

• (I + ∆)−1 = I

And hence

˜ Φx ˜ Φu

• =

 Φx Φu

• (I + ∆)−1

Satisfying nominal constraints results in true system responses:

⇥ zI − ˆ A −ˆ B ⇤ Φx Φu

• = I

Note that if

 xt ut

• =

t

X

k=1

Φx[k] Φu[k]

• et−k

Key to formulation: Write (x,u) as linear function of disturbance Push robustness into cost.

minimize

Φ

sup

k∆Ak2✏A, k∆Bk2✏B

• "

Q

1 2

R

1 2

# Φx Φu

• 2

H2

s.t. ⇥ zI − (ˆ A + ∆A) −(ˆ B + ∆B) ⇤  Φx Φu

• = I

minimize

Φ

sup

k∆Ak2✏A, k∆Bk2✏B

• "

Q

1 2

R

1 2

# Φx Φu

• (I + ∆)1
• 2

H2

s.t. ⇥ zI − ˆ A −ˆ B ⇤  Φx Φu

• = I

SLS Formulation of Robust LQR

• Approximately solvable by SDP for fixed 훾
• Binary search over 훾 to find optimal solution

minimize

γ∈[0,1) 1 1−γ

min

Φx,Φu

• "

Q

1 2

R

1 2

# Φx Φu

• H2

s.t. ⇥ zI − ˆ A −ˆ B ⇤ Φx Φu

• = I
• ✏AΦx

✏BΦu

• H∞

≤

γ √ 2

 xt ut

• =

t

X

k=1

Φx[k] Φu[k]

• et−k

Key to formulation: Write (x,u) as linear function of disturbance

SLS Formulation of Robust LQR

• Solvable by SDP for fixed 훾
• Binary search over 훾 to find optimal solution
• Optimal controller is K=-ZX-1

minimizeX,Z,W,γ

1 (1−γ)2 {Trace(QW11) + Trace(RW22)}

subject to   X X Z∗ X W11 W12 Z W21 W22   ⌫ 0     X I ˆ AX + ˆ BZ (ˆ AX + ˆ BZ)∗ X ✏AX ✏BZ∗ ✏AX ↵2I ✏BZ (1 ↵)2I     ⌫ 0 .

Why robust?

Slightly unstable system, system ID tends to think some nodes are stable

xt+1 =   1.01 0.01 0.01 1.01 0.01 0.01 1.01   xt +   1 1 1   ut + et

Certainty equivalence may yield unstable controller Robust synthesis yields stable controller

Model-free performs worse than model-based

Adaptive LQR

What is the optimal exploration/exploitation scheme? Oracle: You can generate one trajectory of length T. Challenge: Build a controller online with smallest error at every time step.

minimize R(T) :=

T

X

t=1

[x∗

t Qxt + u∗ t Rut − J?]

minimize

u

lim

T→∞ E

h

1 T

PT

t=1 x∗ t Qxt + u∗ t Rut

i s.t. xt+1 = Axt + But + et

time T1 T2 Epoch 1 Epoch 2 [Dean, Mania, Matni, Recht, Tu, NIPS 2018]

SLS for Adaptive LQR

(ˆ A(i), ˆ B(i)) = arg min

(A,B)

X

t∈Ei

kxt+1 Axt Butk2

K(i) = RobustSLS ⇣ ˆ A(i), ˆ B(i), ✏(i)

A , ✏(i) B

⌘

u(i) = K(i)x(i) + η(i)

1. 2. 3.

At every Ti, do:

probing noise (shrinks with T)

Sharp bounds from time-series data: Explore vs. exploit:

[Simchowitz, Mania, Tu, Jordan, Recht, COLT 2018]

OLS Model Mismatch Excitation

• ˆ

A − A ˆ B − B

• = ˜

O ✓ 1 σηT1/2 ◆

Set ηt ∼ N

• 0, σ2

ηI

• <latexit sha1_base64="mJmRUZg9ozuXuma0qOyKuEkI7SA=">ACrXicbVFtaxNBEN6cb7W+NWPflkMQgql3hXBfiwqFCkomkLuesxt5m7LN3dO3bnpOHIX/HX+FW/+2/cTSOYxIGFh+d5ZmZnpmiUdBTHv3vRrdt37t7bur/94OGjxzv93Sdnrm6twJGoVW0vCnCopMERSVJ40VgEXSg8L67eBv38G1ona/OVZg1mGiojSymAPJX3j1IkyImnTmqeaqCpANV9mqcKSxryeD8olY8+C4P+UeWlNaS/vD+KDeBF8EyRLMGDLOM13e1k6qUWr0ZBQ4Nw4iRvKOrAkhcL5dto6bEBcQYVjDw1odFm3GHOX3hmwsva+meIL9h/MzrQzs104Z1hCLeuBfJ/2ril8ijrpGlaQiNuGpWt4lTzsC8+kRYFqZkHIKz0f+ViChYE+a2udFnUblCsTNJdt0aKeoJrKJrsuBJh6RBmjBV914qxb+Acfwk7Piv6sGefhOVpLc/ok/ndnbMPuDJOvr3wRnhweJx59fDY7fLE+zxZ6x52zIEvaHbMP7JSNmGDf2Q/2k/2KXkajKI0ub6xRb5nzlK1EVP0BoWPWCQ=</latexit><latexit sha1_base64="mJmRUZg9ozuXuma0qOyKuEkI7SA=">ACrXicbVFtaxNBEN6cb7W+NWPflkMQgql3hXBfiwqFCkomkLuesxt5m7LN3dO3bnpOHIX/HX+FW/+2/cTSOYxIGFh+d5ZmZnpmiUdBTHv3vRrdt37t7bur/94OGjxzv93Sdnrm6twJGoVW0vCnCopMERSVJ40VgEXSg8L67eBv38G1ona/OVZg1mGiojSymAPJX3j1IkyImnTmqeaqCpANV9mqcKSxryeD8olY8+C4P+UeWlNaS/vD+KDeBF8EyRLMGDLOM13e1k6qUWr0ZBQ4Nw4iRvKOrAkhcL5dto6bEBcQYVjDw1odFm3GHOX3hmwsva+meIL9h/MzrQzs104Z1hCLeuBfJ/2ril8ijrpGlaQiNuGpWt4lTzsC8+kRYFqZkHIKz0f+ViChYE+a2udFnUblCsTNJdt0aKeoJrKJrsuBJh6RBmjBV914qxb+Acfwk7Piv6sGefhOVpLc/ok/ndnbMPuDJOvr3wRnhweJx59fDY7fLE+zxZ6x52zIEvaHbMP7JSNmGDf2Q/2k/2KXkajKI0ub6xRb5nzlK1EVP0BoWPWCQ=</latexit><latexit sha1_base64="mJmRUZg9ozuXuma0qOyKuEkI7SA=">ACrXicbVFtaxNBEN6cb7W+NWPflkMQgql3hXBfiwqFCkomkLuesxt5m7LN3dO3bnpOHIX/HX+FW/+2/cTSOYxIGFh+d5ZmZnpmiUdBTHv3vRrdt37t7bur/94OGjxzv93Sdnrm6twJGoVW0vCnCopMERSVJ40VgEXSg8L67eBv38G1ona/OVZg1mGiojSymAPJX3j1IkyImnTmqeaqCpANV9mqcKSxryeD8olY8+C4P+UeWlNaS/vD+KDeBF8EyRLMGDLOM13e1k6qUWr0ZBQ4Nw4iRvKOrAkhcL5dto6bEBcQYVjDw1odFm3GHOX3hmwsva+meIL9h/MzrQzs104Z1hCLeuBfJ/2ril8ijrpGlaQiNuGpWt4lTzsC8+kRYFqZkHIKz0f+ViChYE+a2udFnUblCsTNJdt0aKeoJrKJrsuBJh6RBmjBV914qxb+Acfwk7Piv6sGefhOVpLc/ok/ndnbMPuDJOvr3wRnhweJx59fDY7fLE+zxZ6x52zIEvaHbMP7JSNmGDf2Q/2k/2KXkajKI0ub6xRb5nzlK1EVP0BoWPWCQ=</latexit><latexit sha1_base64="mJmRUZg9ozuXuma0qOyKuEkI7SA=">ACrXicbVFtaxNBEN6cb7W+NWPflkMQgql3hXBfiwqFCkomkLuesxt5m7LN3dO3bnpOHIX/HX+FW/+2/cTSOYxIGFh+d5ZmZnpmiUdBTHv3vRrdt37t7bur/94OGjxzv93Sdnrm6twJGoVW0vCnCopMERSVJ40VgEXSg8L67eBv38G1ona/OVZg1mGiojSymAPJX3j1IkyImnTmqeaqCpANV9mqcKSxryeD8olY8+C4P+UeWlNaS/vD+KDeBF8EyRLMGDLOM13e1k6qUWr0ZBQ4Nw4iRvKOrAkhcL5dto6bEBcQYVjDw1odFm3GHOX3hmwsva+meIL9h/MzrQzs104Z1hCLeuBfJ/2ril8ijrpGlaQiNuGpWt4lTzsC8+kRYFqZkHIKz0f+ViChYE+a2udFnUblCsTNJdt0aKeoJrKJrsuBJh6RBmjBV914qxb+Acfwk7Piv6sGefhOVpLc/ok/ndnbMPuDJOvr3wRnhweJx59fDY7fLE+zxZ6x52zIEvaHbMP7JSNmGDf2Q/2k/2KXkajKI0ub6xRb5nzlK1EVP0BoWPWCQ=</latexit>

[Dean, Mania, Matni, Recht, Tu, NIPS 2018]

˜ O ✓T1/2 ση ◆

+

˜ O

• σ2

ηT

• <latexit sha1_base64="LbVLliSnU3UmoMS+kx6NX37RZcU=">ACnicbVHbatAEF2rt9S9OeljX5ag0lSKHQPoY2tH1ISErjxGCpZrQayUt2V2J3VGKEv6Ff09f2O/o3WTku1HYHFg7nzGXnTFop6SgM/3SCO3fv3X+w87D76PGTp896u3sXrqytwJEoVWnHKThU0uCIJCkcVxZBpwov06sPrX75Ha2TpTmneYWJhsLIXAogT017Qx6TVBk2p4tYU4DHjtZaJjGSPDtgJ/z2MpiRsPutNcP98Nl8G0QrUCfreJsutJ4qwUtUZDQoFzkyisKGnAkhQKF924dliBuICJx4a0OiSZrnTgr/yTMbz0vpniC/Zfysa0M7NdeozNdDMbWot+T9tUlP+LmkqWpCI24H5bXiVPLWIJ5Ji4LU3AMQVvq/cjEDC4K8jWtTlr0rFGubNe1kaLMcINVdE0WPOmQNEjTbtV8krxr2AcP25N/qv6tq08OJKFJPf62N/KDLeS/UGiTfu3wcXBfuTxlzf9w/er0+ywF+wlG7CIvWH7DM7YyMm2A/2k/1ivwMefAxOgtPb1KCzqnO1iIY3wBYQ9AX</latexit><latexit sha1_base64="LbVLliSnU3UmoMS+kx6NX37RZcU=">ACnicbVHbatAEF2rt9S9OeljX5ag0lSKHQPoY2tH1ISErjxGCpZrQayUt2V2J3VGKEv6Ff09f2O/o3WTku1HYHFg7nzGXnTFop6SgM/3SCO3fv3X+w87D76PGTp896u3sXrqytwJEoVWnHKThU0uCIJCkcVxZBpwov06sPrX75Ha2TpTmneYWJhsLIXAogT017Qx6TVBk2p4tYU4DHjtZaJjGSPDtgJ/z2MpiRsPutNcP98Nl8G0QrUCfreJsutJ4qwUtUZDQoFzkyisKGnAkhQKF924dliBuICJx4a0OiSZrnTgr/yTMbz0vpniC/Zfysa0M7NdeozNdDMbWot+T9tUlP+LmkqWpCI24H5bXiVPLWIJ5Ji4LU3AMQVvq/cjEDC4K8jWtTlr0rFGubNe1kaLMcINVdE0WPOmQNEjTbtV8krxr2AcP25N/qv6tq08OJKFJPf62N/KDLeS/UGiTfu3wcXBfuTxlzf9w/er0+ywF+wlG7CIvWH7DM7YyMm2A/2k/1ivwMefAxOgtPb1KCzqnO1iIY3wBYQ9AX</latexit><latexit sha1_base64="LbVLliSnU3UmoMS+kx6NX37RZcU=">ACnicbVHbatAEF2rt9S9OeljX5ag0lSKHQPoY2tH1ISErjxGCpZrQayUt2V2J3VGKEv6Ff09f2O/o3WTku1HYHFg7nzGXnTFop6SgM/3SCO3fv3X+w87D76PGTp896u3sXrqytwJEoVWnHKThU0uCIJCkcVxZBpwov06sPrX75Ha2TpTmneYWJhsLIXAogT017Qx6TVBk2p4tYU4DHjtZaJjGSPDtgJ/z2MpiRsPutNcP98Nl8G0QrUCfreJsutJ4qwUtUZDQoFzkyisKGnAkhQKF924dliBuICJx4a0OiSZrnTgr/yTMbz0vpniC/Zfysa0M7NdeozNdDMbWot+T9tUlP+LmkqWpCI24H5bXiVPLWIJ5Ji4LU3AMQVvq/cjEDC4K8jWtTlr0rFGubNe1kaLMcINVdE0WPOmQNEjTbtV8krxr2AcP25N/qv6tq08OJKFJPf62N/KDLeS/UGiTfu3wcXBfuTxlzf9w/er0+ywF+wlG7CIvWH7DM7YyMm2A/2k/1ivwMefAxOgtPb1KCzqnO1iIY3wBYQ9AX</latexit><latexit sha1_base64="LbVLliSnU3UmoMS+kx6NX37RZcU=">ACnicbVHbatAEF2rt9S9OeljX5ag0lSKHQPoY2tH1ISErjxGCpZrQayUt2V2J3VGKEv6Ff09f2O/o3WTku1HYHFg7nzGXnTFop6SgM/3SCO3fv3X+w87D76PGTp896u3sXrqytwJEoVWnHKThU0uCIJCkcVxZBpwov06sPrX75Ha2TpTmneYWJhsLIXAogT017Qx6TVBk2p4tYU4DHjtZaJjGSPDtgJ/z2MpiRsPutNcP98Nl8G0QrUCfreJsutJ4qwUtUZDQoFzkyisKGnAkhQKF924dliBuICJx4a0OiSZrnTgr/yTMbz0vpniC/Zfysa0M7NdeozNdDMbWot+T9tUlP+LmkqWpCI24H5bXiVPLWIJ5Ji4LU3AMQVvq/cjEDC4K8jWtTlr0rFGubNe1kaLMcINVdE0WPOmQNEjTbtV8krxr2AcP25N/qv6tq08OJKFJPf62N/KDLeS/UGiTfu3wcXBfuTxlzf9w/er0+ywF+wlG7CIvWH7DM7YyMm2A/2k/1ivwMefAxOgtPb1KCzqnO1iIY3wBYQ9AX</latexit>

σ2 = CηT− 1

3

[Dean, Mania, Matni, Recht, Tu, NIPS 2018]

minimize

u

lim

T→∞ E

h

1 T

PT

t=1 x∗ t Qxt + u∗ t Rut

i s.t. xt+1 = Axt + But + et Fxt ≤ h ∀t, et (x0 given)

Robust Dynamics Robustness to constraints [Dean, Tu, Matni, Recht 2018]

Safe exploration

minimize

1 1−γ

• "

Q

1 2

R

1 2

# Φx Φu

• H2

s.t. ⇥ zI ˆ A ˆ B ⇤ Φx Φu

• = I
• ✏A,2Φx

✏B,2Φu

• H∞



γ √ 2 ,

• ✏A,∞Φx

✏B,∞Φu

• L1

 ⌧ FjΦxx0 +

σe 1−τ kFjΦx[t : 1]k1  bj 8j, t

50000 100000 number of datapoints 10−3 10−2 errors

(a) Estimation Errors

εA,∞ εB,∞ 10−1 1/rx 0.5 1.0 1.5 2.0 2.5 σu

(b) Safety vs. Exploration

ε∞=0.1 ε∞=0.075 ε∞=0.05 ε∞=0.025

[Dean, Tu, Matni, Recht 2018]

Safe exploration

Robust Dynamics Robustness to constraints minimize

1 1−γ

• "

Q

1 2

R

1 2

# Φx Φu

• H2

s.t. ⇥ zI ˆ A ˆ B ⇤ Φx Φu

• = I
• ✏A,2Φx

✏B,2Φu

• H∞



γ √ 2 ,

• ✏A,∞Φx

✏B,∞Φu

• L1

 ⌧ FjΦxx0 +

σe 1−τ kFjΦx[t : 1]k1  bj 8j, t

Enables exploration with safety: u = Kx + η

Probing Robustly Synthesized

Next Steps

• Nonlinear models and constraints via learned ILQR.
• Learning about uncertain environments.
• Model mismatch: what happens when the model is

wrong? Improper learning.

• Implementing in test-beds.

So far…

• Model based methods seem to perform better than

model free ones in theory and practice.

• The field needs more baselines!
• Simple algorithms seem to be surprisingly competitive.
• Analysis of time series is harder than it appears.

References

• argmin.net
• “On the Sample Complexity of the Linear Quadratic Regulator.” S. Dean, H. Mania, N. Matni,
• B. Recht, and S. Tu. arXiv:1710.01688
• “Non-asymptotic Analysis of Robust Control from Coarse-grained Identification.” S. Tu, R.

Boczar, A. Packard, and B. Recht. arXiv:1707.04791

• “Least-squares Temporal Differencing for the Linear Quadratic Regulator” S. Tu and B. Recht.

In submission to ICML 2018. arXiv:1712.08642

• “Learning without Mixing.” H. Mania, B. Recht, M. Simchowitz, and S. Tu. In submission to

COLT 2018. arXiv:1802.08334

• “Simple random search provides a competitive approach to reinforcement learning.” H.

Mania, A. Guy, and B. Recht. arXiv:1803.07055

• “Regret Bounds for Robust Adaptive Control of the Linear Quadratic Regulator.” S. Dean, H.

Mania, N. Matni, B. Recht, and S. Tu. arXiv:1805.09388

• “A Tour of Reinforcement Learning: The View from Continuous Control.” B. Recht.

arXiv:1806.09460

https://people.eecs.berkeley.edu/~brecht/publications.html