Module 15 POMDP Bounds CS 886 Sequential Decision Making and - - PowerPoint PPT Presentation

▶

Feb 14, 2024 273 likes •404 views

Module 15 POMDP Bounds CS 886 Sequential Decision Making and Reinforcement Learning University of Waterloo Bounds POMDP algorithms typically find approximations to optimal value function or optimal policy Need some performance

SLIDE 1

Module 15 POMDP Bounds

CS 886 Sequential Decision Making and Reinforcement Learning University of Waterloo

SLIDE 2

Bounds

POMDP algorithms typically find approximations

to optimal value function or optimal policy

– Need some performance guarantees

Lower bounds on 𝑊∗

– 𝑊𝜌 for any policy 𝜌 – Point-based value iteration

Upper bounds on 𝑊∗

– QMDP – Fast-informed bound – Finite Belief-State MDP

SLIDE 3

Lower Bounds

Lower bounds are easy to obtain
For any policy 𝜌, 𝑊𝜌 is a lower bound

since 𝑊𝜌 𝑐 ≤ 𝑊∗ 𝑐 ∀𝜌, 𝑐

The main issue is to evaluate a policy 𝜌

SLIDE 4

Point-based Value Iteration

Theorem: If 𝑊

0 is a lower bound, then the value

functions 𝑊

𝑜 produced by point-based value

iteration at each iteration 𝑜 are lower bounds.

Proof by induction

– Base case: pick 𝑊

0 to be a lower bound

– Inductive assumption: 𝑊

𝑜 𝑐 ≤ 𝑊∗ 𝑐 ∀𝑐

– Induction:

Let Τ𝑜+1 be the set of 𝛽-vectors for some set 𝐶 of beliefs
Let Τ𝑜+1

∗

be the set of 𝛽-vectors for all beliefs

Hence 𝑊

𝑜+1 𝑐 = max 𝛽∈Τ𝑜+1 𝛽(𝑐) ≤ max 𝛽∈Τ𝑜+1

∗

𝛽 𝑐 ≤ 𝑊∗(𝑐)

SLIDE 5

Upper Bounds

Idea: make decision based on more information

than normally available to obtain higher value than optimal.

POMDP: states are hidden
MDP: states are observable
Hence 𝑊

𝑁𝐸𝑄 ≥ 𝑊 𝑄𝑃𝑁𝐸𝑄

SLIDE 6

QMDP Algorithm

Derive upper bound from MDP Q-function by

allowing the state to be observable

Policy: 𝑡𝑢 → 𝑏𝑢

QMDP(POMDP) Solve MDP to find 𝑅𝑁𝐸𝑄 𝑅𝑁𝐸𝑄 𝑡, 𝑏 = 𝑆 𝑡, 𝑏 + 𝛿 Pr 𝑡′ 𝑡, 𝑏 max

𝑏′ 𝑅𝑁𝐸𝑄(𝑡′, 𝑏′) 𝑡′

Let 𝑊 𝑐 = max

𝑏

𝑐 𝑡 𝑅𝑁𝐸𝑄(𝑡, 𝑏)

𝑡

Return 𝑊

SLIDE 7

Fast Informed Bound

QMDP upper bound is too loose

– Actions depend on current state (too informative)

Tighter upper bound: fast Informed bound (FIB)

– Actions depend on previous state (less informative)

𝑊

𝑁𝐸𝑄 ≥ 𝑊 𝐺𝐽𝐶 ≥ 𝑊∗

SLIDE 8

FIB Algorithm

Derive upper bound by allowing the previous

state to be observable

Policy: 𝑡𝑢−1, 𝑏𝑢−1, 𝑝𝑢 → 𝑏𝑢

FIB(POMDP) Find 𝑅𝐺𝐽𝐶 by value iteration 𝑅𝐺𝐽𝐶 𝑡, 𝑏 = 𝑆 𝑡, 𝑏 + 𝛿

max

𝑏′

Pr 𝑡′ 𝑡, 𝑏 Pr 𝑝′ 𝑡′, 𝑏

𝑡′

𝑅𝐺𝐽𝐶(𝑡′, 𝑏′)

𝑝′

Let 𝑊 𝑐 = max

𝑏

𝑐 𝑡 𝑅𝐺𝐽𝐶(𝑡, 𝑏)

𝑡

Return 𝑊

SLIDE 9

FIB Analysis

Theorem: 𝑊

𝑁𝐸𝑄 ≥ 𝑊 𝐺𝐽𝐶 ≥ 𝑊∗

Proof:

1) 𝑅𝑁𝐸𝑄 𝑡, 𝑏 = 𝑆 𝑡, 𝑏 + 𝛿 Pr 𝑡′ 𝑡, 𝑏 max

𝑏′ 𝑅 𝑡′, 𝑏′ 𝑡′

= 𝑆 𝑡, 𝑏 + 𝛿 Pr 𝑡′ 𝑡, 𝑏 Pr 𝑝′ 𝑡′, 𝑏 max

𝑏′ 𝑅(𝑡′, 𝑏′) 𝑡′𝑝′

≥ 𝑆 𝑡, 𝑏 + 𝛿 max

𝑏′

Pr 𝑡′ 𝑡, 𝑏 Pr 𝑝′ 𝑡′, 𝑏

𝑡′

𝑅(𝑡′, 𝑏′)

𝑝′

= 𝑅𝐺𝐽𝐶(𝑡, 𝑏)

2) 𝑊

𝐺𝐽𝐶 ≥ 𝑊∗ since 𝑊 𝐺𝐽𝐶 is based on observing the

previous state (too informative)

SLIDE 10

Finite Belief-State MDP

Belief state MDP: all beliefs are treated as states

𝑊∗ 𝑐 = max

𝑏

𝑅∗(𝑐, 𝑏)

QMDP and FIB: value of each interior belief is

interpolated: i.e., 𝑊 𝑐 = max

𝑏

𝑐 𝑡 𝑅𝐺𝐽𝐶(𝑡, 𝑏)

𝑡

Idea: retain subset of beliefs

– Interpolate value of remaining beliefs

SLIDE 11

Finite Belief-State MDP

Belief state MDP

𝑅 𝑐, 𝑏 = 𝑆 𝑐, 𝑏 + 𝛿 Pr 𝑝′ 𝑐, 𝑏

𝑝′

max

𝑏′ 𝑅(𝑐𝑏,𝑝, 𝑏′)

Let 𝐶 be a subset of representative beliefs
Approximate 𝑅(𝑐𝑏,𝑝, 𝑏′) with lowest interpolation

– Linear program 𝑅 𝑐𝑏,𝑝, 𝑏′ = min

𝑑

𝑑𝑐𝑅 𝑐, 𝑏′

𝑐∈𝐶

such that 𝑑𝑐 = 1

𝑐

and 𝑑𝑐 ≥ 0 ∀𝑐

SLIDE 12

Finite Belief-State MDP Algorithm

Derive upper bound by interpolating values

based on a finite subset of values

FiniteBeliefStateMDP(POMDP) Find 𝑅𝐶 by value iteration 𝑅𝐶 𝑐, 𝑏 = 𝑆 𝑐, 𝑏 + 𝛿

Pr 𝑝′ 𝑐, 𝑏 max

𝑏′ 𝑅𝐶(𝑐𝑏𝑝′, 𝑏′) 𝑝′

∀𝑐 ∈ 𝐶, 𝑏

where 𝑅𝐶 𝑐𝑏𝑝′, 𝑏′

= min

𝑑

𝑑𝑐𝑅𝐶(𝑐, 𝑏′)

𝑐∈𝐶

such that 𝑑𝑐

𝑐∈𝐶

= 1 and 𝑑𝑐 ≥ 0 ∀𝑐 ∈ 𝐶

Let 𝑊 𝑐 = max

𝑏

𝑐 𝑡 𝑅𝐶(𝑡, 𝑏)

𝑡