[PPT] - Module 8 Linear Programming CS 886 Sequential Decision Making and PowerPoint Presentation

SLIDE 1

Module 8 Linear Programming

CS 886 Sequential Decision Making and Reinforcement Learning University of Waterloo

SLIDE 2

2

Policy Optimization

Value and policy iteration

– Iterative algorithms that implicitly solve an optimization problem

Can we explicitly write down this optimization

problem?

– Yes, it can be formulated as a linear program

SLIDE 3

3

Primal Linear Program

Variables: 𝑊 𝑡 ∀𝑡
Objective: min 𝑥(𝑡)𝑊(𝑡)

𝑡

where 𝑥(𝑡) is a weight assigned to state 𝑡

Constraints:

𝑊 𝑡 ≥ 𝑆 𝑡, 𝑏 + 𝛿 Pr 𝑡′ 𝑡, 𝑏 𝑊 𝑡′ ∀𝑡, 𝑏

𝑡′

primalLP(MDP)

min

𝑊 𝑥(𝑡)𝑊(𝑡) 𝑡

subject to 𝑊 𝑡 ≥ 𝑆 𝑡, 𝑏 + 𝛿 Pr 𝑡′ 𝑡, 𝑏 𝑊 𝑡′ ∀𝑡, 𝑏

𝑡′

return 𝑊

SLIDE 4

4

Objective

Why do we minimize a weighted combination of

the values? Shouldn’t we maximize value?

Value functions 𝑊 that satisfy the constraints are

upper bounds on the optimal value function 𝑊∗ 𝑊 𝑡 ≥ 𝑊∗ 𝑡 ∀𝑡

Minimizing value ensures that we choose the

lowest upper bound

min

V 𝑊(𝑡) = 𝑊∗ 𝑡 ∀𝑡

SLIDE 5

5

Upper bound

Theorem: Value functions 𝑊 that satisfy

𝑊 𝑡 ≥ 𝑆 𝑡, 𝑏 + 𝛿 Pr 𝑡′ 𝑡, 𝑏 𝑊 𝑡′

𝑡′

∀𝑡, 𝑏 are upper bounds on the optimal value function 𝑊∗ 𝑊 𝑡 ≥ 𝑊∗ 𝑡 ∀𝑡

Proof:

– Since 𝑊 𝑡 ≥ 𝑆 𝑡, 𝑏 + 𝛿 Pr 𝑡′ 𝑡, 𝑏 𝑊 𝑡′

𝑡′

∀𝑡, 𝑏 – Then 𝑊 𝑡 ≥ max

𝑏

𝑆 𝑡, 𝑏 + 𝛿 Pr 𝑡′ 𝑡, 𝑏 𝑊 𝑡′

𝑡′

∀𝑡 = 𝐼∗(𝑊)(𝑡) ∀𝑡 – Furthermore 𝑊 ≥ 𝐼∗ 𝑊 ≥ 𝐼∗(𝐼∗ 𝑊 ≥ ⋯ ≥ 𝐼∗ ∞ 𝑊 = 𝑊∗

SLIDE 6

6

Weight function (initial state)

How do we choose the weight function?
If the policy always starts in the same initial state

𝑡0, then set 𝑥 𝑡 = 1 𝑡 = 𝑡0

therwise
This ensures that 𝑥 𝑡 𝑊 𝑡 = 𝑊∗(𝑡0)

𝑡

SLIDE 7

7

Weight function (any state)

If the policy may start in any state, then assign a

positive weight to each state, i.e. 𝑥 𝑡 > 0 ∀𝑡

This ensures that 𝑊 is minimized at each 𝑡 and

therefore 𝑊 𝑡 = 𝑊∗ 𝑡 ∀𝑡

The magnitude of the weight doesn’t matter

when the LP is solved exactly. We will revisit the choice of 𝑥(𝑡) when we discuss approximate linear programming.

SLIDE 8

8

Optimal Policy

Linear program finds 𝑊∗
We can extract 𝜌∗ from 𝑊∗ as usual:

𝜌∗ 𝑡 ← 𝑏𝑠𝑕𝑛𝑏𝑦𝑏 𝑆 𝑡, 𝑏 + 𝛿 Pr 𝑡′ 𝑡, 𝑏 𝑊∗(𝑡′)

𝑡′

Or check the active constraints

– For each 𝑡, check which 𝑏∗ leads to equality 𝑊 𝑡 = 𝑆 𝑡, 𝑏∗ + 𝛿 Pr 𝑡′ 𝑡, 𝑏∗ 𝑊(𝑡′)

𝑡′

𝑊 𝑡 ≥ 𝑆 𝑡, 𝑏 + 𝛿 Pr 𝑡′ 𝑡, 𝑏

𝑡′

𝑊 𝑡′ ∀𝑏 – Set 𝜌∗ 𝑡 ← 𝑏∗

SLIDE 9

9

Direct Policy Optimization

The optimal solution to the primal linear program

is 𝑊∗, but we still have to extract 𝜌∗

Could we directly optimize 𝜌?

– Yes, by considering the dual linear program

SLIDE 10

10

Dual Linear Program

Variables: y 𝑡, 𝑏 ∀𝑡, 𝑏

– frequency of each 𝑡, 𝑏 -pair (proportional to 𝜌)

Objective: max

𝑧

𝑧 𝑡, 𝑏 𝑆(𝑡, 𝑏)

𝑡,𝑏

Constraints:

𝑧 𝑡′, 𝑏′ = 𝑐 𝑡′ + 𝛿 Pr (𝑡′|𝑡, 𝑏)𝑧 𝑡, 𝑏

𝑡,𝑏 𝑏′

dualLP(MDP)

max

𝑧

𝑧 𝑡, 𝑏 𝑆(𝑡, 𝑏)

𝑡,𝑏

subject to 𝑧 𝑡′, 𝑏′ = 𝑐 𝑡′ + 𝛿 Pr (𝑡′|𝑡, 𝑏)𝑧 𝑡, 𝑏

𝑡,𝑏

∀𝑡

𝑏′

𝑧 𝑡, 𝑏 ≥ 0 ∀𝑡, 𝑏 Let 𝜌 𝑏|𝑡 = Pr 𝑏 𝑡 = 𝑧(𝑡, 𝑏)/ 𝑧(𝑡, 𝑏)

𝑏

return 𝜌

SLIDE 11

11

Duality

For every primal linear

program in the form min

𝑦 𝑑𝑈𝑦

s. t. 𝐵𝑦 ≥ 𝑐
There is an equivalent dual

linear program in the form max

𝑧

𝑐𝑈𝑧

s. t. 𝐵𝑈𝑧 = 𝑑 and 𝑧 ≥ 0
Where min

𝑦

𝑑𝑈𝑦 = max

𝑧

𝑐𝑈𝑧

Interpretation: 𝑑 = 𝑥 𝑦 = 𝑊 𝑧 ∝ 𝜌 𝐵 = 𝐽 − 𝛿𝑈𝑏 ∀𝑏 𝑐 = [𝑆𝑏]∀𝑏

SLIDE 12

12

State Frequency

Let 𝑔(𝑡) be the frequency of 𝑡 under policy 𝜌.

0 step: 𝑔

0 𝑡 = 𝑥(𝑡)

1 step: 𝑔

1 𝑡′ = 𝑥 𝑡′ + 𝛿 Pr

(𝑡′|𝑡, 𝜌 𝑡 )𝑥 𝑡

𝑡

2 steps: 𝑔

2 𝑡′′ = 𝑥 𝑡′′ + 𝛿 Pr

(𝑡′′|𝑡′, 𝜌 𝑡′ )𝑥 𝑡′

𝑡′

+𝛿2 Pr 𝑡′′ 𝑡′, 𝜌 𝑡′ Pr 𝑡′ 𝑡, 𝜌 𝑡 𝑥(𝑡)

𝑡,𝑡′

… n steps:

𝑔

𝑜 𝑡 𝑜

= 𝑥 𝑡 𝑜 + 𝛿 Pr 𝑡 𝑜 𝑡 𝑜−1 , 𝜌 𝑡 𝑜−1

𝑡 𝑜−1

𝑔

𝑜−1(𝑡 𝑜−1 )

∞ steps: 𝑔 𝑡′ = 𝑥 𝑡′ + 𝛿 Pr 𝑡′ 𝑡, 𝜌(𝑡)

𝑡

𝑔(𝑡)

SLIDE 13

13

State-Action Frequency

Let 𝑧 𝑡, 𝑏 be the state-action frequency

𝑧 𝑡, 𝑏 = 𝜌 𝑏|𝑡 𝑔 𝑡 where 𝜌 𝑏 𝑡 = Pr 𝑏 𝑡 is a stochastic policy

Then the following equations are equivalent

𝑔 𝑡′ = 𝑥 𝑡′ + 𝛿 Pr 𝑡′ 𝑡, 𝜌(𝑡)

𝑡

𝑔(𝑡) ⇔ 𝜌(𝑏′|𝑡′)

𝑏′

𝑔𝜌 𝑡′ = 𝑥 𝑡′ + Pr 𝑡′ 𝑡, 𝑏 𝜌 𝑏|𝑡 𝑔𝜌(𝑡)

𝑡

⇔ 𝑧(𝑡′, 𝑏′)

𝑏′

= 𝑥 𝑡′ + Pr 𝑡′ 𝑡, 𝑏 𝑧(𝑡, 𝑏)

𝑡

Constraint of dual LP

SLIDE 14

14

Policy

We can recover 𝜌 from 𝑧.

𝑧 𝑡, 𝑏 = 𝜌 𝑏 𝑡 𝑔 𝑡 (by definition) 𝜌 𝑏 𝑡 =

𝑧 𝑡,𝑏 𝑔 𝑡 (isolate 𝜌)

𝜌 𝑏 𝑡 =

𝑧 𝑡,𝑏 𝑧 𝑡,𝑏

𝑏

(by definition)

𝜌 may be stochastic
Actions with non-zero probability are necessarily
ptimal

SLIDE 15

15

Objective

Duality theory guarantees that the objectives of

the primal and dual LPs are equal max

y

𝑧 𝑡, 𝑏 𝑆 𝑡, 𝑏 = min

𝑊 𝑥(𝑡) 𝑡

𝑊(𝑡)

𝑡,𝑏

This means that

𝑧 𝑡, 𝑏 𝑆 𝑡, 𝑏

𝑡,𝑏

implicitly measures the value of the optimal policy.

SLIDE 16

16

Solution Algorithms

Two broad classes of algorithms:

– Simplex (corner search) – Interior point methods (interior iterative methods)

Polynomial complexity (MDP is in P, not NP)
Many packages for linear programming