ts tr r P - - PDF document

❈❙✷✸✹ ◆♦t❡s ✲ ▲❡❝t✉r❡ ✸ ▼♦❞❡❧ ❋r❡❡ P♦❧✐❝② ❊✈❛❧✉❛t✐♦♥✿ P♦❧✐❝② ❊✈❛❧✉❛t✐♦♥ ❲✐t❤♦✉t ❑♥♦✇✐♥❣ ❍♦✇ t❤❡ ❲♦r❧❞ ❲♦r❦s

❨♦✉❦♦✇ ❍♦♠♠❛✱ ❊♠♠❛ ❇r✉♥s❦✐❧❧ ▼❛r❝❤ ✷✵✱ ✷✵✶✽

✹ ▼♦❞❡❧✲❋r❡❡ P♦❧✐❝② ❊✈❛❧✉❛t✐♦♥

■♥ t❤❡ ♣r❡✈✐♦✉s ❧❡❝t✉r❡✱ ✇❡ ❜❡❣❛♥ ❜② ❞✐s❝✉ss✐♥❣ t❤r❡❡ ♣r♦❜❧❡♠ ❢♦r♠✉❧❛t✐♦♥s ♦❢ ✐♥❝r❡❛s✐♥❣ ❝♦♠♣❧❡①✐t②✱ ✇❤✐❝❤ ✇❡ r❡❝❛♣ ❜❡❧♦✇✳ ✶✳ ❆ ▼❛r❦♦✈ ♣r♦❝❡ss ✭▼P✮ ✐s ❛ st♦❝❤❛st✐❝ ♣r♦❝❡ss ❛✉❣♠❡♥t❡❞ ✇✐t❤ t❤❡ ▼❛r❦♦✈ ♣r♦♣❡rt②✳ ✷✳ ❆ ▼❛r❦♦✈ r❡✇❛r❞ ♣r♦❝❡ss ✭▼❘P✮ ✐s ❛ ▼❛r❦♦✈ ♣r♦❝❡ss ✇✐t❤ r❡✇❛r❞s ❛t ❡❛❝❤ t✐♠❡ st❡♣ ❛♥❞ t❤❡ ❛❝❝✉♠✉❧❛t✐♦♥ ♦❢ ❞✐s❝♦✉♥t❡❞ r❡✇❛r❞s✱ ❝❛❧❧❡❞ ✈❛❧✉❡s✳ ✸✳ ❆ ▼❛r❦♦✈ ❞❡❝✐s✐♦♥ ♣r♦❝❡ss ✭▼❉P✮ ✐s ❛ ▼❛r❦♦✈ r❡✇❛r❞ ♣r♦❝❡ss ❛✉❣♠❡♥t❡❞ ✇✐t❤ ❝❤♦✐❝❡s✱ ♦r ❛❝t✐♦♥s✱ ❛t ❡❛❝❤ st❛t❡✳ ■♥ t❤❡ s❡❝♦♥❞ ❤❛❧❢ ♦❢ ❧❛st ❧❡❝t✉r❡✱ ✇❡ ❞✐s❝✉ss❡❞ t✇♦ ♠❡t❤♦❞s ❢♦r ❡✈❛❧✉❛t✐♥❣ ❛ ❣✐✈❡♥ ♣♦❧✐❝② ✐♥ ❛♥ ▼❉P ❛♥❞ t❤r❡❡ ♠❡t❤♦❞s ❢♦r ✜♥❞✐♥❣ t❤❡ ♦♣t✐♠❛❧ ♣♦❧✐❝② ♦❢ ❛♥ ▼❉P✳ ❚❤❡ t✇♦ ♠❡t❤♦❞s ❢♦r ♣♦❧✐❝② ❡✈❛❧✉❛t✐♦♥ ✇❡r❡ ❞✐r❡❝t❧② s♦❧✈✐♥❣ ✈✐❛ ❛ ❧✐♥❡❛r s②st❡♠ ♦❢ ❡q✉❛t✐♦♥s ❛♥❞ ❞②♥❛♠✐❝ ♣r♦❣r❛♠♠✐♥❣✳ ❚❤❡ t❤r❡❡ ♠❡t❤♦❞s ❢♦r ❝♦♥tr♦❧ ✇❡r❡ ❜r✉t❡ ❢♦r❝❡ ♣♦❧✐❝② s❡❛r❝❤✱ ♣♦❧✐❝② ✐t❡r❛t✐♦♥ ❛♥❞ ✈❛❧✉❡ ✐t❡r❛t✐♦♥✳ ■♠♣❧✐❝✐t ✐♥ ❛❧❧ ♦❢ t❤❡s❡ ♠❡t❤♦❞s ✇❛s t❤❡ ❛ss✉♠♣t✐♦♥ t❤❛t ✇❡ ❦♥♦✇ ❜♦t❤ t❤❡ r❡✇❛r❞s ❛♥❞ ♣r♦❜❛❜✐❧✐t✐❡s ❢♦r ❡✈❡r② tr❛♥s✐t✐♦♥✳ ❍♦✇❡✈❡r✱ ✐♥ ♠❛♥② ❝❛s❡s✱ s✉❝❤ ✐♥❢♦r♠❛t✐♦♥ ✐s ♥♦t r❡❛❞✐❧② ❛✈❛✐❧❛❜❧❡ t♦ ✉s✱ ✇❤✐❝❤ ♥❡❝❡ss✐t❛t❡s ♠♦❞❡❧✲❢r❡❡ ❛❧❣♦r✐t❤♠s✳ ■♥ t❤✐s s❡t ♦❢ ❧❡❝t✉r❡s ♥♦t❡s✱ ✇❡ ✇✐❧❧ ❜❡ ❞✐s❝✉ss✐♥❣ ♠♦❞❡❧✲❢r❡❡ ♣♦❧✐❝② ❡✈❛❧✉❛t✐♦♥✳ ❚❤❛t ✐s✱ ✇❡ ✇✐❧❧ ❜❡ ❣✐✈❡♥ ❛ ♣♦❧✐❝② ❛♥❞ ✇✐❧❧ ❛tt❡♠♣t t♦ ❧❡❛r♥ t❤❡ ✈❛❧✉❡ ♦❢ t❤❛t ♣♦❧✐❝② ✇✐t❤♦✉t ❧❡✈❡r❛❣✐♥❣ ❦♥♦✇❧❡❞❣❡ ♦❢ t❤❡ r❡✇❛r❞s ♦r tr❛♥s✐t✐♦♥ ♣r♦❜❛❜✐❧✐t✐❡s✳ ■♥ ♣❛rt✐❝✉❧❛r✱ ✇❡ ✇✐❧❧ ♥♦t ❜❡ ❞✐s❝✉ss✐♥❣ ❤♦✇ t♦ ✐♠♣r♦✈❡ ♦✉r ♣♦❧✐❝✐❡s ✐♥ t❤❡ ♠♦❞❡❧✲❢r❡❡ ❝❛s❡ ✉♥t✐❧ ♥❡①t ❧❡❝t✉r❡✳

✹✳✶ ◆♦t❛t✐♦♥ ❘❡❝❛♣

❇❡❢♦r❡ ❞✐✈✐♥❣ ✐♥t♦ s♦♠❡ ♠❡t❤♦❞s ❢♦r ♠♦❞❡❧✲❢r❡❡ ♣♦❧✐❝② ❡✈❛❧✉❛t✐♦♥✱ ✇❡✬❧❧ ✜rst r❡❝❛♣ s♦♠❡ ♦❢ t❤❡ ♥♦t❛t✐♦♥ s✉rr♦✉♥❞✐♥❣ ▼❉P✬s ❢r♦♠ ❧❛st ❧❡❝t✉r❡ t❤❛t ✇❡✬❧❧ ♥❡❡❞ ✐♥ t❤✐s ❧❡❝t✉r❡✳ ❘❡❝❛❧❧ t❤❛t ✇❡ ❞❡✜♥❡ t❤❡ r❡t✉r♥ ♦❢ ❛♥ ▼❘P ❛s t❤❡ ❞✐s❝♦✉♥t❡❞ s✉♠ ♦❢ r❡✇❛r❞s st❛rt✐♥❣ ❢r♦♠ t✐♠❡ st❡♣ t ❛♥❞ ❡♥❞✐♥❣ ❛t ❤♦r✐③♦♥ H✱ ✇❤❡r❡ H ♠❛② ❜❡ ✐♥✜♥✐t❡✳ ▼❛t❤❡♠❛t✐❝❛❧❧②✱ t❤✐s t❛❦❡s t❤❡ ❢♦r♠ Gt =

H−1

• i=t

γi−tri, ✭✶✮ ❢♦r 0 ≤ t ≤ H − 1✱ ✇❤❡r❡ 0 < γ ≤ 1 ✐s t❤❡ ❞✐s❝♦✉♥t ❢❛❝t♦r ❛♥❞ ri ✐s t❤❡ r❡✇❛r❞ ❛t t✐♠❡ st❡♣ i✳ ❋♦r ❛♥ ▼❉P✱ t❤❡ r❡t✉r♥ Gt ✐s ❞❡✜♥❡❞ ✐❞❡♥t✐❝❛❧❧②✱ ❛♥❞ t❤❡ r❡✇❛r❞s ri ❛r❡ ❣❡♥❡r❛t❡❞ ❜② ❢♦❧❧♦✇✐♥❣ ♣♦❧✐❝② π(a|s)✳ ✶

❚❤❡ st❛t❡ ✈❛❧✉❡ ❢✉♥❝t✐♦♥ V π(s) ✐s t❤❡ ❡①♣❡❝t❡❞ r❡t✉r♥ ❢r♦♠ st❛rt✐♥❣ ❛t st❛t❡ s ✉♥❞❡r st❛t✐♦♥❛r② ♣♦❧✐❝② π✳ ▼❛t❤❡♠❛t✐❝❛❧❧②✱ ✇❡ ❝❛♥ ❡①♣r❡ss t❤✐s ❛s V π(s) = Eπ[Gt|st = s] ✭✷✮ = Eπ H−1

• i=t

γi−tri|st = s

• .

✭✸✮ ❚❤❡ st❛t❡✲❛❝t✐♦♥ ✈❛❧✉❡ ❢✉♥❝t✐♦♥ Qπ(s, a) ✐s t❤❡ ❡①♣❡❝t❡❞ r❡t✉r♥ ❢r♦♠ st❛rt✐♥❣ ✐♥ st❛t❡ s✱ t❛❦✐♥❣ ❛❝t✐♦♥ a✱ ❛♥❞ t❤❡♥ ❢♦❧❧♦✇✐♥❣ st❛t✐♦♥❛r② ♣♦❧✐❝② π ❢♦r ❛❧❧ tr❛♥s✐t✐♦♥s t❤❡r❡❛❢t❡r✳ ▼❛t❤❡♠❛t✐❝❛❧❧②✱ ✇❡ ❝❛♥ ❡①♣r❡ss t❤✐s ❛s Qπ(s, a) = Eπ[Gt|st = s, at = a] ✭✹✮ = Eπ H−1

• i=t

γi−tri|st = s, at = a

• .

✭✺✮ ❚❤r♦✉❣❤♦✉t t❤✐s ❧❡❝t✉r❡✱ ✇❡ ✇✐❧❧ ❛ss✉♠❡ ❛♥ ✐♥✜♥✐t❡ ❤♦r✐③♦♥ ❛s ✇❡❧❧ ❛s st❛t✐♦♥❛r② r❡✇❛r❞s✱ tr❛♥s✐t✐♦♥ ♣r♦❜❛❜✐❧✐t✐❡s ❛♥❞ ♣♦❧✐❝✐❡s✳ ❚❤✐s ❛❧❧♦✇s ✉s t♦ ❤❛✈❡ t✐♠❡✲✐♥❞❡♣❡♥❞❡♥t st❛t❡ ❛♥❞ st❛t❡✲❛❝t✐♦♥ ✈❛❧✉❡ ❢✉♥❝✲ t✐♦♥s✱ ❛s ❞❡r✐✈❡❞ ❧❛st ❧❡❝t✉r❡✳ ❚❤❡r❡ ✐s ♦♥❡ ♥❡✇ ❞❡✜♥✐t✐♦♥ t❤❛t ✇❡ ✇✐❧❧ ✉s❡ ✐♥ t❤✐s ❧❡❝t✉r❡✱ ✇❤✐❝❤ ✐s t❤❡ ❞❡✜♥✐t✐♦♥ ♦❢ ❛ ❤✐st♦r②✳ ❉❡✜♥✐t✐♦♥ ✹✳✶✳ ❚❤❡ ❤✐st♦r② ✐s t❤❡ ♦r❞❡r❡❞ t✉♣❧❡ ♦❢ st❛t❡s✱ ❛❝t✐♦♥s ❛♥❞ r❡✇❛r❞s t❤❛t ❛♥ ❛❣❡♥t ❡①✲ ♣❡r✐❡♥❝❡s✳ ■♥ ❡♣✐s♦❞✐❝ ❞♦♠❛✐♥s✱ ✇❡ ✇✐❧❧ ✉s❡ t❤❡ ✇♦r❞ ❡♣✐s♦❞❡ ✐♥t❡r❝❤❛♥❣❡❛❜❧② ✇✐t❤ ❤✐st♦r②✳ ❲❤❡♥ ❝♦♥s✐❞❡r✐♥❣ ♠❛♥② ✐♥t❡r❛❝t✐♦♥s✱ ✇❡ ✇✐❧❧ ✐♥❞❡① t❤❡ ❤✐st♦r✐❡s ✐♥ t❤❡ ❢♦❧❧♦✇✐♥❣ ♠❛♥♥❡r✿ t❤❡ jt❤ ❤✐st♦r② ✐s hj = (sj,1, aj,1, rj,1, sj,2, aj,2, rj,2, . . . , sj,Lj), ✇❤❡r❡ Lj ✐s t❤❡ ❧❡♥❣t❤ ♦❢ t❤❡ ✐♥t❡r❛❝t✐♦♥✱ ❛♥❞ sj,t, aj,t, rj,t ❛r❡ t❤❡ st❛t❡✱ ❛❝t✐♦♥ ❛♥❞ r❡✇❛r❞ ❛t t✐♠❡ st❡♣ t ✐♥ ❤✐st♦r② j✱ r❡s♣❡❝t✐✈❡❧②✳

✹✳✷ ❉②♥❛♠✐❝ Pr♦❣r❛♠♠✐♥❣

❘❡❝❛❧❧ ❛❧s♦ ❢r♦♠ ❧❛st ❧❡❝t✉r❡ t❤❡ ❞②♥❛♠✐❝ ♣r♦❣r❛♠♠✐♥❣ ❛❧❣♦r✐t❤♠ t♦ ❝❛❧❝✉❧❛t❡ t❤❡ ✈❛❧✉❡ ♦❢ ❛♥ ✐♥✜♥✐t❡ ❤♦r✐③♦♥ ▼❉P ✇✐t❤ γ < 1 ✉♥❞❡r ♣♦❧✐❝② π✱ ✇❤✐❝❤ ✇❡ r❡✇r✐t❡ ❤❡r❡ ❢♦r ❝♦♥✈❡♥✐❡♥❝❡ ❛s ❆❧❣♦r✐t❤♠ ✶✳ ❆❧❣♦r✐t❤♠ ✶ ■t❡r❛t✐✈❡ ❛❧❣♦r✐t❤♠ t♦ ❝❛❧❝✉❧❛t❡ ▼❉P ✈❛❧✉❡ ❢✉♥❝t✐♦♥ ❢♦r ❛ st❛t✐♦♥❛r② ♣♦❧✐❝② π

✶✿ ♣r♦❝❡❞✉r❡ P♦❧✐❝② ❊✈❛❧✉❛t✐♦♥✭M, π, ǫ✮ ✷✿

❋♦r ❛❧❧ st❛t❡s s ∈ S✱ ❞❡✜♥❡ Rπ(s) =

a∈A π(a|s)R(s, a)

✸✿

❋♦r ❛❧❧ st❛t❡s s, s′ ∈ S✱ ❞❡✜♥❡ P π(s′|s) =

a∈A π(a|s)P(s′|s, a)

✹✿

❋♦r ❛❧❧ st❛t❡s s ∈ S✱ V0(s) ← 0

✺✿

k ← 0

✻✿

✇❤✐❧❡ k = 0 ♦r ||Vk − Vk−1||∞ > ǫ ❞♦

✼✿

k ← k + 1

✽✿

❋♦r ❛❧❧ st❛t❡s s ∈ S✱ Vk(s) = Rπ(s) + γ

s′∈S P π(s′|s)Vk−1(s′)

✾✿

r❡t✉r♥ Vk ❲r✐tt❡♥ ✐♥ t❤✐s ❢♦r♠✱ ✇❡ ❝❛♥ t❤✐♥❦ ♦❢ Vk ✐♥ ❛ ❢❡✇ ✇❛②s✳ ❋✐rst✱ Vk(s) ✐s t❤❡ ❡①❛❝t ✈❛❧✉❡ ♦❢ ❢♦❧❧♦✇✐♥❣ ♣♦❧✐❝② π ❢♦r k ❛❞❞✐t✐♦♥❛❧ tr❛♥s✐t✐♦♥s✱ st❛rt✐♥❣ ❛t st❛t❡ s✳ ❙❡❝♦♥❞✱ ❢♦r ❧❛r❣❡ k✱ ♦r ✇❤❡♥ ❆❧❣♦r✐t❤♠ ✶ t❡r♠✐♥❛t❡s✱ Vk(s) ✐s ❛♥ ❡st✐♠❛t❡ ♦❢ t❤❡ tr✉❡✱ ✐♥✜♥✐t❡ ❤♦r✐③♦♥ ✈❛❧✉❡ V π(s) ♦❢ st❛t❡ s✳ ❲❡ ❝❛♥ ❛❞❞✐t✐♦♥❛❧❧② ❡①♣r❡ss t❤❡ ❜❡❤❛✈✐♦r ♦❢ t❤✐s ❛❧❣♦r✐t❤♠ ✈✐❛ ❛ ❜❛❝❦✉♣ ❞✐❛❣r❛♠✱ ✇❤✐❝❤ ✐s s❤♦✇♥ ✐♥ ❋✐❣✉r❡ ✶✳ ❚❤✐s ❜❛❝❦✉♣ ❞✐❛❣r❛♠ ✐s r❡❛❞ t♦♣✲❞♦✇♥ ✇✐t❤ ✇❤✐t❡ ❝✐r❝❧❡s r❡♣r❡s❡♥t✐♥❣ st❛t❡s✱ ❜❧❛❝❦ ❝✐r❝❧❡s ✷

❋✐❣✉r❡ ✶✿ ❇❛❝❦✉♣ ❞✐❛❣r❛♠ ❢♦r t❤❡ ❞②♥❛♠✐❝ ♣r♦❣r❛♠♠✐♥❣ ♣♦❧✐❝② ❡✈❛❧✉❛t✐♦♥ ❛❧❣♦r✐t❤♠✳ r❡♣r❡s❡♥t✐♥❣ ❛❝t✐♦♥s ❛♥❞ ❛r❝s r❡♣r❡s❡♥t✐♥❣ t❛❦✐♥❣ t❤❡ ❡①♣❡❝t❛t✐♦♥✳ ❚❤❡ ❞✐❛❣r❛♠ s❤♦✇s t❤❡ ❜r❛♥❝❤✐♥❣ ❡✛❡❝t ♦❢ st❛rt✐♥❣ ❛t st❛t❡ s ❛t t❤❡ t♦♣ ❛♥❞ tr❛♥s✐t✐♦♥✐♥❣ t✇♦ t✐♠❡ st❡♣s ❛s ✇❡ ♠♦✈❡ ❞♦✇♥ t❤❡ ❞✐❛❣r❛♠✳ ■t ❢✉rt❤❡r♠♦r❡ s❤♦✇s ❤♦✇ ❛❢t❡r st❛rt✐♥❣ ❛t st❛t❡ s ❛♥❞ t❛❦✐♥❣ ❛♥ ❛❝t✐♦♥ ✉♥❞❡r ♣♦❧✐❝② π✱ ✇❡ t❛❦❡ t❤❡ ❡①♣❡❝t❛t✐♦♥ ♦✈❡r t❤❡ ✈❛❧✉❡ ♦❢ t❤❡ ♥❡①t st❛t❡✳ ■♥ ❞②♥❛♠✐❝ ♣r♦❣r❛♠♠✐♥❣✱ ✇❡ ❜♦♦tstr❛♣✱ ♦r ❡st✐♠❛t❡✱ t❤❡ ✈❛❧✉❡ ♦❢ t❤❡ ♥❡①t st❛t❡ ✉s✐♥❣ ♦✉r ❝✉rr❡♥t ❡st✐♠❛t❡✱ Vk−1(s′)✳

✹✳✸ ▼♦♥t❡ ❈❛r❧♦ ❖♥ P♦❧✐❝② ❊✈❛❧✉❛t✐♦♥

❲❡ ♥♦✇ ❞❡s❝r✐❜❡ ♦✉r ✜rst ♠♦❞❡❧✲❢r❡❡ ♣♦❧✐❝② ❡✈❛❧✉❛t✐♦♥ ❛❧❣♦r✐t❤♠ ✇❤✐❝❤ ✉s❡s ❛ ♣♦♣✉❧❛r ❝♦♠♣✉t❛t✐♦♥❛❧ ♠❡t❤♦❞ ❝❛❧❧❡❞ t❤❡ ▼♦♥t❡ ❈❛r❧♦ ♠❡t❤♦❞✳ ❲❡ ✜rst ✇❛❧❦ t❤r♦✉❣❤ ❛♥ ❡①❛♠♣❧❡ ♦❢ t❤❡ ▼♦♥t❡ ❈❛r❧♦ ♠❡t❤♦❞ ♦✉ts✐❞❡ t❤❡ ❝♦♥t❡①t ♦❢ r❡✐♥❢♦r❝❡♠❡♥t ❧❡❛r♥✐♥❣✱ t❤❡♥ ❞✐s❝✉ss t❤❡ ♠❡t❤♦❞ ♠♦r❡ ❣❡♥❡r❛❧❧②✱ ❛♥❞ ✜♥❛❧❧② ❛♣♣❧② ▼♦♥t❡ ❈❛r❧♦ t♦ r❡✐♥❢♦r❝❡♠❡♥t ❧❡❛r♥✐♥❣✳ ❲❡ ❡♠♣❤❛s✐③❡ ❤❡r❡ t❤❛t t❤✐s ♠❡t❤♦❞ ♦♥❧② ✇♦r❦s ✐♥ ❡♣✐s♦❞✐❝ ❡♥✈✐r♦♥♠❡♥ts✱ ❛♥❞ ✇❡✬❧❧ s❡❡ ✇❤② t❤✐s ✐s ❛s ✇❡ ❡①❛♠✐♥❡ t❤❡ ❛❧❣♦r✐t❤♠ ♠♦r❡ ❝❛r❡❢✉❧❧② ✐♥ t❤✐s s❡❝t✐♦♥✳ ❙✉♣♣♦s❡ ✇❡ ✇❛♥t t♦ ❡st✐♠❛t❡ ❤♦✇ ❧♦♥❣ t❤❡ ❝♦♠♠✉t❡ ❢r♦♠ ②♦✉r ❤♦✉s❡ t♦ ❙t❛♥❢♦r❞✬s ❝❛♠♣✉s ✇✐❧❧ t❛❦❡ t♦❞❛②✳ ❙✉♣♣♦s❡ ✇❡ ❛❧s♦ ❤❛✈❡ ❛❝❝❡ss t♦ ❛ ❝♦♠♠✉t❡ s✐♠✉❧❛t♦r ✇❤✐❝❤ ♠♦❞❡❧s ♦✉r ✉♥❝❡rt❛✐♥t② ♦❢ ❤♦✇ ❜❛❞ t❤❡ tr❛✣❝ ✇✐❧❧ ❜❡✱ t❤❡ ✇❡❛t❤❡r✱ ❝♦♥str✉❝t✐♦♥ ❞❡❧❛②s✱ ❛♥❞ ♦t❤❡r ✈❛r✐❛❜❧❡s✱ ❛s ✇❡❧❧ ❛s ❤♦✇ t❤❡s❡ ✈❛r✐❛❜❧❡s ✐♥t❡r❛❝t ✇✐t❤ ❡❛❝❤ ♦t❤❡r✳ ❖♥❡ ✇❛② t♦ ❡st✐♠❛t❡ t❤❡ ❡①♣❡❝t❡❞ ❝♦♠♠✉t❡ t✐♠❡ ✐s t♦ s✐♠✉❧❛t❡ ♦✉r ❝♦♠♠✉t❡ ♠❛♥② t✐♠❡s ♦♥ t❤❡ s✐♠✉❧❛t♦r ❛♥❞ t❤❡♥ t❛❦❡ ❛♥ ❛✈❡r❛❣❡ ♦✈❡r t❤❡ s✐♠✉❧❛t❡❞ ❝♦♠♠✉t❡ t✐♠❡s✳ ❚❤✐s ✐s ❝❛❧❧❡❞ ❛ ▼♦♥t❡ ❈❛r❧♦ ❡st✐♠❛t❡ ♦❢ ♦✉r ❝♦♠♠✉t❡ t✐♠❡✳ ■♥ ❣❡♥❡r❛❧✱ ✇❡ ❣❡t t❤❡ ▼♦♥t❡ ❈❛r❧♦ ❡st✐♠❛t❡ ♦❢ s♦♠❡ q✉❛♥t✐t② ❜② ♦❜s❡r✈✐♥❣ ♠❛♥② ✐t❡r❛t✐♦♥s ♦❢ ❤♦✇ t❤❛t q✉❛♥t✐t② ✐s ❣❡♥❡r❛t❡❞ ❡✐t❤❡r ✐♥ r❡❛❧ ❧✐❢❡ ♦r ✈✐❛ s✐♠✉❧❛t✐♦♥ ❛♥❞ t❤❡♥ ❛✈❡r❛❣✐♥❣ ♦✈❡r t❤❡ ♦❜s❡r✈❡❞ q✉❛♥t✐t✐❡s✳ ❇② t❤❡ ❧❛✇ ♦❢ ❧❛r❣❡ ♥✉♠❜❡rs✱ t❤✐s ❛✈❡r❛❣❡ ❝♦♥✈❡r❣❡s t♦ t❤❡ ❡①♣❡❝t❛t✐♦♥ ♦❢ t❤❡ q✉❛♥t✐t②✳ ■♥ t❤❡ ❝♦♥t❡①t ♦❢ r❡✐♥❢♦r❝❡♠❡♥t ❧❡❛r♥✐♥❣✱ t❤❡ q✉❛♥t✐t② ✇❡ ✇❛♥t t♦ ❡st✐♠❛t❡ ✐s V π(s)✱ ✇❤✐❝❤ ✐s t❤❡ ❛✈❡r❛❣❡ ♦❢ r❡t✉r♥s Gt ✉♥❞❡r ♣♦❧✐❝② π st❛rt✐♥❣ ❛t st❛t❡ s✳ ❲❡ ❝❛♥ t❤✉s ❣❡t ❛ ▼♦♥t❡ ❈❛r❧♦ ❡st✐♠❛t❡ ♦❢ V π(s) t❤r♦✉❣❤ t❤r❡❡ st❡♣s✿ ✶✳ ❊①❡❝✉t❡ ❛ r♦❧❧♦✉t ♦❢ ♣♦❧✐❝② π ✉♥t✐❧ t❡r♠✐♥❛t✐♦♥ ♠❛♥② t✐♠❡s ✷✳ ❘❡❝♦r❞ t❤❡ r❡t✉r♥s Gt t❤❛t ✇❡ ♦❜s❡r✈❡ ✇❤❡♥ st❛rt✐♥❣ ❛t st❛t❡ s ✸✳ ❚❛❦❡ ❛♥ ❛✈❡r❛❣❡ ♦❢ t❤❡ ✈❛❧✉❡s ✇❡ ❣❡t ❢♦r Gt t♦ ❡st✐♠❛t❡ V π(s)✳ ❚❤❡ ❜❛❝❦✉♣ ❞✐❛❣r❛♠ ❢♦r ▼♦♥t❡ ❈❛r❧♦ ♣♦❧✐❝② ❡✈❛❧✉❛t✐♦♥ ❝❛♥ ❜❡ s❡❡♥ ✐♥ ❋✐❣✉r❡ ✷✳ ❚❤❡ ♥❡✇ ❜❧✉❡ ❧✐♥❡ ✐♥❞✐❝❛t❡s t❤❛t ✇❡ s❛♠♣❧❡ ❛♥ ❡♥t✐r❡ ❡♣✐s♦❞❡ ✉♥t✐❧ t❡r♠✐♥❛t✐♦♥ st❛rt✐♥❣ ❛t st❛t❡ s✳ ❚❤❡r❡ ❛r❡ t✇♦ ❢♦r♠s ♦❢ ▼♦♥t❡ ❈❛r❧♦ ♦♥ ♣♦❧✐❝② ❡✈❛❧✉❛t✐♦♥✱ ✇❤✐❝❤ ❛r❡ ❞✐✛❡r❡♥t✐❛t❡❞ ❜② ✇❤❡t❤❡r ✇❡ t❛❦❡ ❛♥ ❛✈❡r❛❣❡ ♦✈❡r ❥✉st t❤❡ ✜rst t✐♠❡ ✇❡ ✈✐s✐t ❛ st❛t❡ ✐♥ ❡❛❝❤ r♦❧❧♦✉t ♦r ❡✈❡r② t✐♠❡ ✇❡ ✈✐s✐t t❤❡ st❛t❡ ✐♥ ❡❛❝❤ ✸

❋✐❣✉r❡ ✷✿ ❇❛❝❦✉♣ ❞✐❛❣r❛♠ ❢♦r t❤❡ ▼♦♥t❡ ❈❛r❧♦ ♣♦❧✐❝② ❡✈❛❧✉❛t✐♦♥ ❛❧❣♦r✐t❤♠✳ r♦❧❧♦✉t✳ ❚❤❡s❡ ❛r❡ ❝❛❧❧❡❞ ❋✐rst✲❱✐s✐t ▼♦♥t❡ ❈❛r❧♦ ❛♥❞ ❊✈❡r②✲❱✐s✐t ▼♦♥t❡ ❈❛r❧♦ ❖♥ P♦❧✐❝② ❊✈❛❧✉❛t✐♦♥✱ r❡s♣❡❝t✐✈❡❧②✳ ▼♦r❡ ❢♦r♠❛❧❧②✱ ✇❡ ❞❡s❝r✐❜❡ t❤❡ ❋✐rst✲❱✐s✐t ▼♦♥t❡ ❈❛r❧♦ ✐♥ ❆❧❣♦r✐t❤♠ ✷ ❛♥❞ t❤❡ ❊✈❡r②✲❱✐s✐t ▼♦♥t❡ ❈❛r❧♦ ✐♥ ❆❧❣♦r✐t❤♠ ✸✳ ❆❧❣♦r✐t❤♠ ✷ ❋✐rst✲❱✐s✐t ▼♦♥t❡ ❈❛r❧♦ P♦❧✐❝② ❊✈❛❧✉❛t✐♦♥

✶✿ ♣r♦❝❡❞✉r❡ ❋✐rst✲❱✐s✐t✲▼♦♥t❡✲❈❛r❧♦✭h1, . . . , hj✮ ✷✿

❋♦r ❛❧❧ st❛t❡s s✱ N(s) ← 0✱ S(s) ← 0✱ V (s) ← 0

✸✿

❢♦r ❡❛❝❤ ❡♣✐s♦❞❡ hj ❞♦

✹✿

❢♦r t = 1, . . . , Lj ❞♦

✺✿

✐❢ sj,t = sj,u ❢♦r u < t t❤❡♥

✻✿

N(sj,t) ← N(sj,t) + 1

✼✿

S(sj,t) ← S(sj,t) + Gj,t

✽✿

V π(sj,t) ← S(sj,t)/N(sj,t)

✾✿

r❡t✉r♥ V π ❆❧❣♦r✐t❤♠ ✸ ❊✈❡r②✲❱✐s✐t ▼♦♥t❡ ❈❛r❧♦ P♦❧✐❝② ❊✈❛❧✉❛t✐♦♥

✶✿ ♣r♦❝❡❞✉r❡ ❊✈❡r②✲❱✐s✐t✲▼♦♥t❡✲❈❛r❧♦✭h1, . . . , hj✮ ✷✿

❋♦r ❛❧❧ st❛t❡s s✱ N(s) ← 0✱ S(s) ← 0✱ V (s) ← 0

✸✿

❢♦r ❡❛❝❤ ❡♣✐s♦❞❡ hj ❞♦

✹✿

❢♦r t = 1, . . . , Lj ❞♦

✺✿

N(sj,t) ← N(sj,t) + 1

✻✿

S(sj,t) ← S(sj,t) + Gj,t

✼✿

V π(sj,t) ← S(sj,t)/N(sj,t)

✽✿

r❡t✉r♥ V π ◆♦t❡ t❤❛t ✐♥ t❤❡ ❜♦❞② ♦❢ t❤❡ ❢♦r ❧♦♦♣ ✐♥ ❆❧❣♦r✐t❤♠s ✷ ❛♥❞ ✸✱ ✇❡ ❝❛♥ r❡♠♦✈❡ ✈❡❝t♦r S ❛♥❞ r❡♣❧❛❝❡ t❤❡ ✉♣❞❛t❡ ❢♦r V π(sj,t) ✇✐t❤ V π(sj,t) ← V π(sj,t) + 1 N(sj,t)(Gj,t − V π(sj,t)). ✭✻✮ ❚❤✐s ✐s ❜❡❝❛✉s❡ t❤❡ ♥❡✇ ❛✈❡r❛❣❡ ✐s t❤❡ ❛✈❡r❛❣❡ ♦❢ N(sj,t) − 1 ♦❢ t❤❡ ♦❧❞ ✈❛❧✉❡s V π(sj,t) ❛♥❞ t❤❡ ♥❡✇ r❡t✉r♥ Gj,t✱ ❣✐✈✐♥❣ ✉s V π(sj,t) × (N(sj,t) − 1) + Gj,t N(sj,t) = V π(sj,t) + 1 N(sj,t)(Gj,t − V π(sj,t)), ✭✼✮ ✹

✇❤✐❝❤ ✐s ♣r❡❝✐s❡❧② t❤❡ ♥❡✇ ❢♦r♠ ♦❢ t❤❡ ✉♣❞❛t❡✳ ❘❡♣❧❛❝✐♥❣

1 N(sj,t) ✇✐t❤ α ✐♥ t❤✐s ♥❡✇ ✉♣❞❛t❡ ❣✐✈❡s ✉s t❤❡ ♠♦r❡ ❣❡♥❡r❛❧ ■♥❝r❡♠❡♥t❛❧ ▼♦♥t❡ ❈❛r❧♦

❖♥ P♦❧✐❝② ❊✈❛❧✉❛t✐♦♥✳ ❆❧❣♦r✐t❤♠s ✹ ❛♥❞ ✺ ❞❡t❛✐❧ t❤✐s ♣r♦❝❡❞✉r❡ ✐♥ t❤❡ ❋✐rst✲❱✐s✐t ❛♥❞ ❊✈❡r②✲❱✐s✐t ❝❛s❡s✱ r❡s♣❡❝t✐✈❡❧②✳ ❆❧❣♦r✐t❤♠ ✹ ■♥❝r❡♠❡♥t❛❧ ❋✐rst✲❱✐s✐t ▼♦♥t❡ ❈❛r❧♦ P♦❧✐❝② ❊✈❛❧✉❛t✐♦♥

✶✿ ♣r♦❝❡❞✉r❡ ■♥❝r❡♠❡♥t❛❧✲❋✐rst✲❱✐s✐t✲▼♦♥t❡✲❈❛r❧♦✭α, h1, . . . , hj✮ ✷✿

❋♦r ❛❧❧ st❛t❡s s✱ N(s) ← 0✱ V (s) ← 0

✸✿

❢♦r ❡❛❝❤ ❡♣✐s♦❞❡ hj ❞♦

✹✿

❢♦r t = 1, . . . , terminal ❞♦

✺✿

✐❢ sj,t = sj,u ❢♦r u < t t❤❡♥

✻✿

N(sj,t) ← N(sj,t) + 1

✼✿

V π(sj,t) ← V π(s) + α(Gj,t − V π(s))

✽✿

r❡t✉r♥ V π ❆❧❣♦r✐t❤♠ ✺ ■♥❝r❡♠❡♥t❛❧ ❊✈❡r②✲❱✐s✐t ▼♦♥t❡ ❈❛r❧♦ P♦❧✐❝② ❊✈❛❧✉❛t✐♦♥

✶✿ ♣r♦❝❡❞✉r❡ ■♥❝r❡♠❡♥t❛❧✲❊✈❡r②✲❱✐s✐t✲▼♦♥t❡✲❈❛r❧♦✭α, h1, . . . , hj✮ ✷✿

❋♦r ❛❧❧ st❛t❡s s✱ N(s) ← 0✱ V (s) ← 0

✸✿

❢♦r ❡❛❝❤ ❡♣✐s♦❞❡ hj ❞♦

✹✿

❢♦r t = 1, . . . , terminal ❞♦

✺✿

N(sj,t) ← N(sj,t) + 1

✻✿

V π(sj,t) ← V π(s) + α(Gj,t − V π(s))

✼✿

r❡t✉r♥ V π ❙❡tt✐♥❣ α =

1 N(sj,t) r❡❝♦✈❡rs t❤❡ ♦r✐❣✐♥❛❧ ▼♦♥t❡ ❈❛r❧♦ ❖♥ P♦❧✐❝② ❊✈❛❧✉❛t✐♦♥ ❛❧❣♦r✐t❤♠s ❣✐✈❡♥ ✐♥ ❆❧❣♦✲

r✐t❤♠s ✷ ❛♥❞ ✸✱ ✇❤✐❧❡ s❡tt✐♥❣ α >

1 N(s) ❣✐✈❡s ❛ ❤✐❣❤❡r ✇❡✐❣❤t t♦ ♥❡✇❡r ❞❛t❛✱ ✇❤✐❝❤ ❝❛♥ ❤❡❧♣ ❧❡❛r♥✐♥❣

✐♥ ♥♦♥✲st❛t✐♦♥❛r② ❞♦♠❛✐♥s✳ ■❢ ✇❡ ❛r❡ ✐♥ ❛ tr✉❧② ▼❛r❦♦✈✐❛♥✲❞♦♠❛✐♥✱ ❊✈❡r②✲❱✐s✐t ▼♦♥t❡ ❈❛r❧♦ ✇✐❧❧ ❜❡ ♠♦r❡ ❞❛t❛ ❡✣❝✐❡♥t ❜❡❝❛✉s❡ ✇❡ ✉♣❞❛t❡ ♦✉r ❛✈❡r❛❣❡ r❡t✉r♥ ❢♦r ❛ st❛t❡ ❡✈❡r② t✐♠❡ ✇❡ ✈✐s✐t t❤❡ st❛t❡✳ ❊①❡r❝✐s❡ ✹✳✶✳ ❘❡❝❛❧❧ ♦✉r ▼❛rs ❘♦✈❡r ▼❉P ❢r♦♠ ❧❛st ❧❡❝t✉r❡✱ s❤♦✇♥ ✐♥ ❋✐❣✉r❡ ✸ ❜❡❧♦✇✳ ❙✉♣♣♦s❡ t❤❛t ♦✉r ❡st✐♠❛t❡ ❢♦r t❤❡ ✈❛❧✉❡ ♦❢ ❡❛❝❤ st❛t❡ ✐s ❝✉rr❡♥t❧② ✵✳ ■❢ ✇❡ ❡①♣❡r✐❡♥❝❡ t❤❡ ❤✐st♦r② h = (S3, TL, +0, S2, TL, +0, S1, TL, +1, terminal), t❤❡♥✿ ✶✳ ❲❤❛t ✐s t❤❡ ✜rst✲✈✐s✐t ▼♦♥t❡ ❈❛r❧♦ ❡st✐♠❛t❡ ♦❢ ❱ ❛t ❡❛❝❤ st❛t❡❄ ✷✳ ❲❤❛t ✐s t❤❡ ❡✈❡r②✲✈✐s✐t ▼♦♥t❡ ❈❛r❧♦ ❡st✐♠❛t❡ ♦❢ ❡❛❝❤ st❛t❡❄ ✸✳ ❲❤❛t ✐s t❤❡ ✐♥❝r❡♠❡♥t❛❧ ✜rst✲✈✐s✐t ▼♦♥t❡ ❈❛r❧♦ ❡st✐♠❛t❡ ♦❢ ❱ ✇✐t❤ α = 2

3❄

✹✳ ❲❤❛t ✐s t❤❡ ✐♥❝r❡♠❡♥t❛❧ ❡✈❡r②✲✈✐s✐t ▼♦♥t❡ ❈❛r❧♦ ❡st✐♠❛t❡ ♦❢ ❱ ✇✐t❤ α = 2

3❄

✹✳✹ ▼♦♥t❡ ❈❛r❧♦ ❖✛ P♦❧✐❝② ❊✈❛❧✉❛t✐♦♥

■♥ t❤❡ s❡❝t✐♦♥ ❛❜♦✈❡✱ ✇❡ ❞✐s❝✉ss❡❞ t❤❡ ❝❛s❡ ✇❤❡r❡ ✇❡ ❛r❡ ❛❜❧❡ t♦ ♦❜t❛✐♥ ♠❛♥② r❡❛❧✐③❛t✐♦♥s ♦❢ Gt ✉♥❞❡r t❤❡ ♣♦❧✐❝② π t❤❛t ✇❡ ✇❛♥t t♦ ❡✈❛❧✉❛t❡✳ ❍♦✇❡✈❡r✱ ✐♥ ♠❛♥② ❝♦st❧② ♦r ❤✐❣❤ st❛❦❡s s✐t✉❛t✐♦♥s✱ ✇❡ ❛r❡♥✬t ❛❜❧❡ t♦ ♦❜t❛✐♥ r♦❧❧♦✉ts ♦❢ Gt ✉♥❞❡r t❤❡ ♣♦❧✐❝② t❤❛t ✇❡ ✇✐s❤ t♦ ❡✈❛❧✉❛t❡✳ ❋♦r ❡①❛♠♣❧❡✱ ✇❡ ♠❛② ❤❛✈❡ ❞❛t❛ ❛ss♦❝✐❛t❡❞ ✇✐t❤ ♦♥❡ ♠❡❞✐❝❛❧ ♣♦❧✐❝②✱ ❜✉t ✇❛♥t t♦ ❞❡t❡r♠✐♥❡ t❤❡ ✈❛❧✉❡ ♦❢ ❛ ❞✐✛❡r❡♥t ♠❡❞✐❝❛❧ ♣♦❧✐❝②✳ ■♥ t❤✐s s❡❝t✐♦♥✱ ✇❡ ❞❡s❝r✐❜❡ ▼♦♥t❡ ❈❛r❧♦ ♦✛ ♣♦❧✐❝② ❡✈❛❧✉❛t✐♦♥✱ ✇❤✐❝❤ ✐s ❛ ♠❡t❤♦❞ ❢♦r ✉s✐♥❣ ❞❛t❛ t❛❦❡♥ ❢r♦♠ ♦♥❡ ♣♦❧✐❝② t♦ ❡✈❛❧✉❛t❡ ❛ ❞✐✛❡r❡♥t ♣♦❧✐❝②✳ ✺

❋✐❣✉r❡ ✸✿ ▼❛rs ❘♦✈❡r ▼❛r❦♦✈ ❉❡❝✐s✐♦♥ Pr♦❝❡ss ✇✐t❤ ❛❝t✐♦♥s ❚r② ▲❡❢t ✭❚▲✮ ❛♥❞ ❚r② ❘✐❣❤t ✭❚❘✮ ✹✳✹✳✶ ■♠♣♦rt❛♥❝❡ ❙❛♠♣❧✐♥❣ ❚❤❡ ❦❡② ✐♥❣r❡❞✐❡♥t ♦❢ ♦✛ ♣♦❧✐❝② ❡✈❛❧✉❛t✐♦♥ ✐s ❛ ♠❡t❤♦❞ ❝❛❧❧❡❞ ✐♠♣♦rt❛♥❝❡ s❛♠♣❧✐♥❣✳ ❚❤❡ ❣♦❛❧ ♦❢ ✐♠♣♦rt❛♥❝❡ s❛♠♣❧✐♥❣ ✐s t♦ ❡st✐♠❛t❡ t❤❡ ❡①♣❡❝t❡❞ ✈❛❧✉❡ ♦❢ ❛ ❢✉♥❝t✐♦♥ f(x) ✇❤❡♥ x ✐s ❞r❛✇♥ ❢r♦♠ ❞✐str✐❜✉t✐♦♥ q ✉s✐♥❣ ♦♥❧② t❤❡ ❞❛t❛ f(x1), . . . , f(xn)✱ ✇❤❡r❡ xi ❛r❡ ❞r❛✇♥ ❢r♦♠ ❛ ❞✐✛❡r❡♥t ❞✐str✐❜✉t✐♦♥ p✳ ■♥ s✉♠♠❛r②✱ ❣✐✈❡♥ q(xi), p(xi), f(xi) ❢♦r 1 ≤ xi, ≤ n✱ ✇❡ ✇♦✉❧❞ ❧✐❦❡ ❛♥ ❡st✐♠❛t❡ ❢♦r Ex∼q[f(x)]✳ ❲❡ ❝❛♥ ❞♦ t❤✐s ✈✐❛ t❤❡ ❢♦❧❧♦✇✐♥❣ ❛♣♣r♦①✐♠❛t✐♦♥✿ Ex∼q[f(x)] =

• x

q(x)f(x)dx ✭✽✮ =

• x

p(x) q(x) p(x)f(x)

• dx

✭✾✮ = Ex∼p q(x) p(x)f(x)

• ✭✶✵✮

≈

n

• i=1

q(xi) p(xi)f(xi)

• .

✭✶✶✮ ❚❤❡ ❧❛st ❡q✉❛t✐♦♥ ❣✐✈❡s ✉s t❤❡ ✐♠♣♦rt❛♥❝❡ s❛♠♣❧✐♥❣ ❡st✐♠❛t❡ ♦❢ f ✉♥❞❡r ❞✐str✐❜✉t✐♦♥ q ✉s✐♥❣ s❛♠♣❧❡s ♦❢ f ✉♥❞❡r ❞✐str✐❜✉t✐♦♥ p✳ ◆♦t❡ t❤❛t t❤❡ ✜rst st❡♣ ♦♥❧② ❤♦❧❞s ✐❢ q(x)f(x) > 0 ✐♠♣❧✐❡s p(x) > 0 ❢♦r ❛❧❧ x✳ ✹✳✹✳✷ ■♠♣♦rt❛♥❝❡ ❙❛♠♣❧✐♥❣ ❢♦r ❖✛ P♦❧✐❝② ❊✈❛❧✉❛t✐♦♥ ❲❡ ♥♦✇ ❛♣♣❧② t❤❡ ❣❡♥❡r❛❧ r❡s✉❧t ♦❢ ✐♠♣♦rt❛♥❝❡ s❛♠♣❧✐♥❣ ❡st✐♠❛t❡s t♦ r❡✐♥❢♦r❝❡♠❡♥t ❧❡❛r♥✐♥❣✳ ■♥ t❤✐s ✐♥st❛♥❝❡✱ ✇❡ ✇❛♥t t♦ ❛♣♣r♦①✐♠❛t❡ t❤❡ ✈❛❧✉❡ ♦❢ st❛t❡ s ✉♥❞❡r ♣♦❧✐❝② π1✱ ❣✐✈❡♥ ❜② V π1(s) = E[Gt|st = s]✱ ✉s✐♥❣ n ❤✐st♦r✐❡s h1, . . . , hn ❣❡♥❡r❛t❡❞ ✉♥❞❡r ♣♦❧✐❝② π2✳ ❯s✐♥❣ t❤❡ ✐♠♣♦rt❛♥❝❡ s❛♠♣❧✐♥❣ ❡st✐♠❛t❡ r❡s✉❧t ❣✐✈❡s ✉s t❤❛t V π1(s) ≈ 1 n

n

• j=1

p(hj|π1, s) p(hj|π2, s)G(hj), ✭✶✷✮ ✇❤❡r❡ G(hj) = Lj−1

t=1

γt−1rj,t ✐s t❤❡ t♦t❛❧ ❞✐s❝♦✉♥t❡❞ s✉♠ ♦❢ r❡✇❛r❞s ❢♦r ❤✐st♦r② hj✳ ✻

◆♦✇✱ ❢♦r ❛ ❣❡♥❡r❛❧ ♣♦❧✐❝② π✱ ✇❡ ❤❛✈❡ t❤❛t t❤❡ ♣r♦❜❛❜✐❧✐t② ♦❢ ❡①♣❡r✐❡♥❝✐♥❣ ❤✐st♦r② hj ✉♥❞❡r ♣♦❧✐❝② π ✐s p(hj|π, s = sj,1) =

Lj−1

• t=1

p(aj,t|sj,t)p(rj,t|sj,t, aj,t)p(sj,t+1|sj,t, aj,t) ✭✶✸✮ =

Lj−1

• t=1

π(aj,t|sj,t)p(rj,t|sj,t, aj,t)p(sj,t+1|sj,t, aj,t), ✭✶✹✮ ✇❤❡r❡ Lj ✐s t❤❡ ❧❡♥❣t❤ ♦❢ t❤❡ jt❤ ❡♣✐s♦❞❡✳ ❚❤❡ ✜rst ❧✐♥❡ ❢♦❧❧♦✇s ❢r♦♠ ❧♦♦❦✐♥❣ ❛t t❤❡ t❤r❡❡ ❝♦♠♣♦♥❡♥ts ♦❢ ❡❛❝❤ tr❛♥s✐t✐♦♥✳ ❚❤❡ ❝♦♠♣♦♥❡♥ts ❛r❡✿ ✶✳ p(aj,t|sj,t) ✲ ♣r♦❜❛❜✐❧✐t② ✇❡ t❛❦❡ ❛❝t✐♦♥ aj,t ❛t st❛t❡ sj,t ✷✳ p(rj,t|sj,t, aj,t) ✲ ♣r♦❜❛❜✐❧✐t② ✇❡ ❡①♣❡r✐❡♥❝❡ r❡✇❛r❞ rj,t ❛❢t❡r t❛❦✐♥❣ ❛❝t✐♦♥ aj,t ✐♥ st❛t❡ sj,t ✸✳ p(sj,t+1|sj,t, aj,t) ✲ ♣r♦❜❛❜✐❧✐t② ✇❡ tr❛♥s✐t✐♦♥ t♦ st❛t❡ sj,t+1 ❛❢t❡r t❛❦✐♥❣ ❛❝t✐♦♥ aj,t ✐♥ st❛t❡ sj,t ◆♦✇✱ ❝♦♠❜✐♥✐♥❣ ♦✉r ✐♠♣♦rt❛♥❝❡ s❛♠♣❧✐♥❣ ❡st✐♠❛t❡ ❢♦r V π1(s) ✇✐t❤ ♦✉r ❞❡❝♦♠♣♦s✐t✐♦♥ ♦❢ t❤❡ ❤✐st♦r② ♣r♦❜❛❜✐❧✐t✐❡s✱ p(hj|π, s = sj,1)✱ ✇❡ ❣❡t t❤❛t V π1(s) ≈ 1 n

n

• j=1

p(hj|π1, s) p(hj|π2, s)G(hj) ✭✶✺✮ = 1 n

n

• j=1

Lj−1

t=1

π1(aj,t|sj,t)p(rj,t|sj,t, aj,t)p(sj,t+1|sj,t, aj,t) Lj−1

t=1

π2(aj,t|sj,t)p(rj,t|sj,t, aj,t)p(sj,t+1|sj,t, aj,t) G(hj) ✭✶✻✮ = 1 n

n

• j=1

G(hj)

Lj−1

• t=1

π1(aj,t|sj,t) π2(aj,t|sj,t). ✭✶✼✮ ◆♦t✐❝❡ ✇❡ ❝❛♥ ♥♦✇ ❡①♣❧✐❝✐t❧② ❡✈❛❧✉❛t❡ t❤❡ ❡①♣r❡ss✐♦♥ ✇✐t❤♦✉t t❤❡ tr❛♥s✐t✐♦♥ ♣r♦❜❛❜✐❧✐t✐❡s ♦r r❡✇❛r❞s s✐♥❝❡ ❛❧❧ ♦❢ t❤❡ t❡r♠s ✐♥✈♦❧✈✐♥❣ ♠♦❞❡❧ ❞②♥❛♠✐❝s ❝❛♥❝❡❧❡❞ ♦✉t ✐♥ t❤❡ s❡❝♦♥❞ st❡♣ ♦❢ t❤❡ ❡q✉❛t✐♦♥✳ ■♥ ♣❛rt✐❝✉❧❛r✱ ✇❡ ❛r❡ ❣✐✈❡♥ t❤❡ ❤✐st♦r✐❡s hj✱ s♦ ✇❡ ❝❛♥ ❝❛❧❝✉❧❛t❡ G(hj) = Lj−1

t=1

γt−1rj,t ✱ ❛♥❞ ✇❡ ❦♥♦✇ t❤❡ t✇♦ ♣♦❧✐❝✐❡s π1 ❛♥❞ π2✱ s♦ ✇❡ ❝❛♥ ❛❧s♦ ❡✈❛❧✉❛t❡ t❤❡ s❡❝♦♥❞ t❡r♠✳

✹✳✺ ❚❡♠♣♦r❛❧ ❉✐✛❡r❡♥❝❡ ✭❚❉✮ ▲❡❛r♥✐♥❣

❙♦ ❢❛r✱ ✇❡ ❤❛✈❡ t✇♦ ♠❡t❤♦❞s ❢♦r ♣♦❧✐❝② ❡✈❛❧✉❛t✐♦♥✿ ❞②♥❛♠✐❝ ♣r♦❣r❛♠♠✐♥❣ ❛♥❞ ▼♦♥t❡ ❈❛r❧♦✳ ❉②♥❛♠✐❝ ♣r♦❣r❛♠♠✐♥❣ ❧❡✈❡r❛❣❡s ❜♦♦tstr❛♣♣✐♥❣ t♦ ❤❡❧♣ ✉s ❣❡t ✈❛❧✉❡ ❡st✐♠❛t❡s ✇✐t❤ ♦♥❧② ♦♥❡ ❜❛❝❦✉♣✳ ❖♥ t❤❡ ♦t❤❡r ❤❛♥❞✱ ▼♦♥t❡ ❈❛r❧♦ s❛♠♣❧❡s ♠❛♥② ❤✐st♦r✐❡s ❢♦r ♠❛♥② tr❛❥❡❝t♦r✐❡s ✇❤✐❝❤ ❢r❡❡s ✉s ❢r♦♠ ✉s✐♥❣ ❛ ♠♦❞❡❧✳ ◆♦✇✱ ✇❡ ✐♥tr♦❞✉❝❡ ❛ ♥❡✇ ❛❧❣♦r✐t❤♠ t❤❛t ❝♦♠❜✐♥❡s ❜♦♦tstr❛♣♣✐♥❣ ✇✐t❤ s❛♠♣❧✐♥❣ t♦ ❣✐✈❡ ✉s ❛ s❡❝♦♥❞ ♠♦❞❡❧✲❢r❡❡ ♣♦❧✐❝② ❡✈❛❧✉❛t✐♦♥ ❛❧❣♦r✐t❤♠✳ ❚♦ s❡❡ ❤♦✇ t♦ ❝♦♠❜✐♥❡ s❛♠♣❧✐♥❣ ✇✐t❤ ❜♦♦tstr❛♣♣✐♥❣✱ ❧❡t✬s ❣♦ ❜❛❝❦ t♦ ♦✉r ✐♥❝r❡♠❡♥t❛❧ ▼♦♥t❡ ❈❛r❧♦ ✉♣❞❛t❡✿ V π(st) ← V π(st) + α(Gt − V π(st)). ✭✶✽✮ ❘❡❝❛❧❧ t❤❛t Gt ✐s t❤❡ r❡t✉r♥ ❛❢t❡r r♦❧❧✐♥❣ ♦✉t t❤❡ ♣♦❧✐❝② ❢r♦♠ t✐♠❡ st❡♣ t t♦ t❡r♠✐♥❛t✐♦♥ st❛rt✐♥❣ ❛t st❛t❡ st✳ ▲❡t✬s ♥♦✇ r❡♣❧❛❝❡ Gt ✇✐t❤ ❛ ❇❡❧❧♠❛♥ ❜❛❝❦✉♣ ❧✐❦❡ ✐♥ ❞②♥❛♠✐❝ ♣r♦❣r❛♠♠✐♥❣✳ ❚❤❛t ✐s✱ ❧❡t✬s r❡♣❧❛❝❡ Gt ✇✐t❤ rt + γV π(st+1)✱ ✇❤❡r❡ rt ✐s ❛ s❛♠♣❧❡ ♦❢ t❤❡ r❡✇❛r❞ ❛t t✐♠❡ st❡♣ t ❛♥❞ V π(st+1) ✐s ♦✉r ❝✉rr❡♥t ❡st✐♠❛t❡ ♦❢ t❤❡ ✈❛❧✉❡ ❛t t❤❡ ♥❡①t st❛t❡✳ ▼❛❦✐♥❣ t❤✐s s✉❜st✐t✉t✐♦♥ ❣✐✈❡s ✉s t❤❡ ❚❉✲❧❡❛r♥✐♥❣ ✉♣❞❛t❡ V π(st) ← V π(st) + α(rt + γV π(st+1) − V π(st)). ✭✶✾✮ ❚❤❡ ❞✐✛❡r❡♥❝❡ δt = rt + γV π(st+1) − V π(st) ✭✷✵✮ ✼

❋✐❣✉r❡ ✹✿ ❇❛❝❦✉♣ ❞✐❛❣r❛♠ ❢♦r t❤❡ ❚❉ ▲❡❛r♥✐♥❣ ♣♦❧✐❝② ❡✈❛❧✉❛t✐♦♥ ❛❧❣♦r✐t❤♠✳ ✐s ❝♦♠♠♦♥❧② r❡❢❡rr❡❞ t♦ ❛s t❤❡ ❚❉ ❡rr♦r✱ ❛♥❞ t❤❡ s❛♠♣❧❡❞ r❡✇❛r❞ ❝♦♠❜✐♥❡❞ ✇✐t❤ t❤❡ ❜♦♦tstr❛♣ ❡st✐♠❛t❡ ♦❢ t❤❡ ♥❡①t st❛t❡ ✈❛❧✉❡✱ rt + γV π(st+1), ✭✷✶✮ ✐s r❡❢❡rr❡❞ t♦ ❛s t❤❡ ❚❉ t❛r❣❡t✳ ❚❤❡ ❢✉❧❧ ❚❉ ❧❡❛r♥✐♥❣ ❛❧❣♦r✐t❤♠ ✐s ❣✐✈❡♥ ✐♥ ❆❧❣♦r✐t❤♠ ✻✳ ❲❡ ❝❛♥ s❡❡ t❤❛t ✉s✐♥❣ t❤✐s ♠❡t❤♦❞✱ ✇❡ ✉♣❞❛t❡ ♦✉r ✈❛❧✉❡ ❢♦r V π(st) ❞✐r❡❝t❧② ❛❢t❡r ✇✐t♥❡ss✐♥❣ t❤❡ tr❛♥s✐t✐♦♥ (st, at, rt, st+1)✳ ■♥ ♣❛rt✐❝✉❧❛r✱ ✇❡ ❞♦♥✬t ♥❡❡❞ t♦ ✇❛✐t ❢♦r t❤❡ ❡♣✐s♦❞❡ t♦ t❡r♠✐♥❛t❡ ❧✐❦❡ ✐♥ ▼♦♥t❡ ❈❛r❧♦✳ ❆❧❣♦r✐t❤♠ ✻ ❚❉ ▲❡❛r♥✐♥❣ t♦ ❡✈❛❧✉❛t❡ ♣♦❧✐❝② π

✶✿ ♣r♦❝❡❞✉r❡ ❚❉▲❡❛r♥✐♥❣✭st❡♣ s✐③❡ α✱ ♥✉♠❜❡r ♦❢ tr❛❥❡❝t♦r✐❡s n✮ ✷✿

❋♦r ❛❧❧ st❛t❡s s✱ V π(s) ← 0

✸✿

✇❤✐❧❡ n > 0 ❞♦

✹✿

❇❡❣✐♥ ❡♣✐s♦❞❡ E ❛t st❛t❡ s

✺✿

✇❤✐❧❡ n > 0 ❛♥❞ ❡♣✐s♦❞❡ E ❤❛s ♥♦t t❡r♠✐♥❛t❡❞ ❞♦

✻✿

a ← ❛❝t✐♦♥ ❛t st❛t❡ s ✉♥❞❡r ♣♦❧✐❝② π

✼✿

❚❛❦❡ ❛❝t✐♦♥ a ✐♥ E ❛♥❞ ♦❜s❡r✈❡ r❡✇❛r❞ r✱ ♥❡①t st❛t❡ s′

✽✿

V π(s) ← V π(s) + α(R + γV π(s′) − V π(s))

✾✿

s ← s′

✶✵✿

r❡t✉r♥ V π ❲❡ ❝❛♥ ❛❣❛✐♥ ❡①❛♠✐♥❡ t❤✐s ❛❧❣♦r✐t❤♠ ✈✐❛ ❛ ❜❛❝❦✉♣ ❞✐❛❣r❛♠ ❛s s❤♦✇♥ ✐♥ ❋✐❣✉r❡ ✹✳ ❍❡r❡✱ ✇❡ s❡❡ ✈✐❛ t❤❡ ❜❧✉❡ ❧✐♥❡ t❤❛t ✇❡ s❛♠♣❧❡ ♦♥❡ tr❛♥s✐t✐♦♥ st❛rt✐♥❣ ❛t s✱ t❤❡♥ ✇❡ ❡st✐♠❛t❡ t❤❡ ✈❛❧✉❡ ♦❢ t❤❡ ♥❡①t st❛t❡ ✈✐❛ ♦✉r ❝✉rr❡♥t ❡st✐♠❛t❡ ♦❢ t❤❡ ♥❡①t st❛t❡ t♦ ❝♦♥str✉❝t ❛ ❢✉❧❧ ❇❡❧❧♠❛♥ ❜❛❝❦✉♣ ❡st✐♠❛t❡✳ ❚❤❡r❡ ✐s ❛❝t✉❛❧❧② ❛♥ ❡♥t✐r❡ s♣❡❝tr✉♠ ♦❢ ✇❛②s ✇❡ ❝❛♥ ❜❧❡♥❞ ▼♦♥t❡ ❈❛r❧♦ ❛♥❞ ❞②♥❛♠✐❝ ♣r♦❣r❛♠♠✐♥❣ ✉s✐♥❣ ❛ ♠❡t❤♦❞ ❝❛❧❧❡❞ ❚❉✭λ✮✳ ❲❤❡♥ λ = 0✱ ✇❡ ❣❡t t❤❡ ❚❉✲❧❡❛r♥✐♥❣ ❢♦r♠✉❧❛t✐♦♥ ❛❜♦✈❡✱ ❤❡♥❝❡ ❣✐✈✐♥❣ ✉s t❤❡ ❛❧✐❛s ❚❉✭✵✮✳ ❲❤❡♥ λ = 1✱ ✇❡ r❡❝♦✈❡r ▼♦♥t❡ ❈❛r❧♦ ♣♦❧✐❝② ❡✈❛❧✉❛t✐♦♥✱ ❞❡♣❡♥❞✐♥❣ ♦♥ t❤❡ ❢♦r♠✉❧❛t✐♦♥ ✉s❡❞✳ ❲❤❡♥ 0 < λ < 1✱ ✇❡ ❣❡t ❛ ❜❧❡♥❞ ♦❢ t❤❡s❡ t✇♦ ♠❡t❤♦❞s✳ ❋♦r ❛ ♠♦r❡ t❤♦r♦✉❣❤ tr❡❛t♠❡♥t ♦❢ ❚❉✭λ✮✱ ✇❡ r❡❢❡r t❤❡ ✐♥t❡r❡st❡❞ r❡❛❞❡r t♦ ❙❡❝t✐♦♥s ✼✳✶ ❛♥❞ ✶✷✳✶✲✶✷✳✺ ♦❢ ❙✉tt♦♥ ❛♥❞ ❇❛rt♦ ❬✶❪ ✇❤✐❝❤ ❞❡t❛✐❧ n✲st❡♣ ❚❉ ❧❡❛r♥✐♥❣ ❛♥❞ ❚❉✭λ✮✴❡❧✐❣✐❜✐❧✐t② tr❛❝❡s✱ r❡s♣❡❝t✐✈❡❧②✳ ❊①❡r❝✐s❡ ✹✳✷✳ ❈♦♥s✐❞❡r ❛❣❛✐♥ t❤❡ ▼❛rs ❘♦✈❡r ❡①❛♠♣❧❡ ✐♥ ❋✐❣✉r❡ ✸✳ ❙✉♣♣♦s❡ t❤❛t ♦✉r ❡st✐♠❛t❡ ❢♦r t❤❡ ✈❛❧✉❡ ♦❢ ❡❛❝❤ st❛t❡ ✐s ❝✉rr❡♥t❧② ✵✳ ■❢ ✇❡ ❡①♣❡r✐❡♥❝❡ t❤❡ ❤✐st♦r② h = (S3, TL, +0, S2, TL, +0, S2, TL, +0, S1, TL, +1, terminal), t❤❡♥✿ ✽

✶✳ ❲❤❛t ✐s t❤❡ ❚❉✭✵✮ ❡st✐♠❛t❡ ♦❢ ❱ ✇✐t❤ α = 1❄ ✷✳ ❲❤❛t ✐s t❤❡ ❚❉✭✵✮ ❡st✐♠❛t❡ ♦❢ ❱ ✇✐t❤ α = 2

3❄

✹✳✻ ❙✉♠♠❛r② ♦❢ ▼❡t❤♦❞s ❉✐s❝✉ss❡❞

■♥ t❤✐s ❧❡❝t✉r❡✱ ✇❡ r❡✲❡①❛♠✐♥❡❞ ♣♦❧✐❝② ❡✈❛❧✉❛t✐♦♥ ✉s✐♥❣ ❞②♥❛♠✐❝ ♣r♦❣r❛♠♠✐♥❣ ❢r♦♠ ❧❛st ❧❡❝t✉r❡✱ ❛♥❞ ✇❡ ✐♥tr♦❞✉❝❡❞ t✇♦ ♥❡✇ ♠❡t❤♦❞s ❢♦r ♣♦❧✐❝② ❡✈❛❧✉❛t✐♦♥✱ ♥❛♠❡❧② ▼♦♥t❡ ❈❛r❧♦ ❡✈❛❧✉❛t✐♦♥ ❛♥❞ ❚❡♠♣♦r❛❧ ❉✐✛❡r❡♥❝❡ ✭❚❉✮ ❧❡❛r♥✐♥❣✳ ❋✐rst✱ ✇❡ ♠♦t✐✈❛t❡❞ t❤❡ ✐♥tr♦❞✉❝t✐♦♥ ♦❢ ▼♦♥t❡ ❈❛r❧♦ ❛♥❞ ❚❉✲▲❡❛r♥✐♥❣ ❜② ♥♦t✐♥❣ t❤❛t ❞②♥❛♠✐❝ ♣r♦✲ ❣r❛♠♠✐♥❣ r❡❧✐❡❞ ♦♥ ❛ ♠♦❞❡❧ ♦❢ t❤❡ ✇♦r❧❞✳ ❚❤❛t ✐s✱ ✇❡ ♥❡❡❞❡❞ t♦ ❢❡❡❞ ♦✉r ❞②♥❛♠✐❝ ♣r♦❣r❛♠♠✐♥❣ ♣♦❧✐❝② ❡✈❛❧✉❛t✐♦♥ ❛❧❣♦r✐t❤♠ ✇✐t❤ t❤❡ r❡✇❛r❞s ❛♥❞ tr❛♥s✐t✐♦♥ ♣r♦❜❛❜✐❧✐t✐❡s ♦❢ t❤❡ ❞♦♠❛✐♥✳ ▼♦♥t❡ ❈❛r❧♦ ❛♥❞ ❚❉✲▲❡❛r♥✐♥❣ ❛r❡ ❜♦t❤ ❢r❡❡ ❢r♦♠ t❤✐s ❝♦♥str❛✐♥t✱ ♠❛❦✐♥❣ t❤❡♠ ♠♦❞❡❧✲❢r❡❡ ♠❡t❤♦❞s✳ ■♥ ▼♦♥t❡ ❈❛r❧♦ ♣♦❧✐❝② ❡✈❛❧✉❛t✐♦♥✱ ✇❡ ❣❡♥❡r❛t❡ ♠❛♥② ❤✐st♦r✐❡s ❛♥❞ t❤❡♥ ❛✈❡r❛❣❡ t❤❡ r❡t✉r♥s ♦✈❡r t❤❡ st❛t❡s t❤❛t ✇❡ ❡♥❝♦✉♥t❡r✳ ■♥ ♦r❞❡r ❢♦r ✉s t♦ ❣❡♥❡r❛t❡ t❤❡s❡ ❤✐st♦r✐❡s ✐♥ ❛ ✜♥✐t❡ ❛♠♦✉♥t ♦❢ t✐♠❡✱ ✇❡ r❡q✉✐r❡ t❤❡ ❞♦♠❛✐♥ t♦ ❜❡ ❡♣✐s♦❞✐❝ ✲ t❤❛t ✐s✱ ✇❡ ♥❡❡❞ t♦ ❡♥s✉r❡ t❤❛t ❡❛❝❤ ❤✐st♦r② t❤❛t ✇❡ ♦❜s❡r✈❡ t❡r♠✐♥❛t❡s✳ ■♥ ❜♦t❤ ❞②♥❛♠✐❝ ♣r♦❣r❛♠♠✐♥❣ ❛♥❞ t❡♠♣♦r❛❧ ❞✐✛❡r❡♥❝❡ ❧❡❛r♥✐♥❣✱ ✇❡ ♦♥❧② ❜❛❝❦✉♣ ♦✈❡r ♦♥❡ tr❛♥s✐t✐♦♥ ✭✇❡ ♦♥❧② ❧♦♦❦ ♦♥❡ st❡♣ ❛❤❡❛❞ ✐♥ t❤❡ ❢✉t✉r❡✮✱ s♦ t❡r♠✐♥❛t✐♦♥ ♦❢ ❤✐st♦r✐❡s ✐s ♥♦t ❛ ❝♦♥❝❡r♥✱ ❛♥❞ ✇❡ ❝❛♥ ❛♣♣❧② t❤❡s❡ ❛❧❣♦r✐t❤♠s t♦ ♥♦♥✲❡♣✐s♦❞✐❝ ❞♦♠❛✐♥s✳ ❖♥ t❤❡ ✢✐♣ s✐❞❡✱ t❤❡ r❡❛s♦♥ ✇❡ ❛r❡ ❛❜❧❡ t♦ ❜❛❝❦✉♣ ♦✈❡r ❥✉st ♦♥❡ tr❛♥s✐t✐♦♥ ✐♥ ❞②♥❛♠✐❝ ♣r♦❣r❛♠♠✐♥❣ ❛♥❞ ❚❉ ❧❡❛r♥✐♥❣ ✐s ❜❡❝❛✉s❡ ✇❡ ❧❡✈❡r❛❣❡ t❤❡ ▼❛r❦♦✈✐❛♥ ❛ss✉♠♣t✐♦♥ ♦❢ t❤❡ ❞♦♠❛✐♥✳ ❋✉rt❤❡r♠♦r❡✱ ■♥❝r❡♠❡♥t❛❧ ▼♦♥t❡ ❈❛r❧♦ ♣♦❧✐❝② ❡✈❛❧✉❛t✐♦♥✱ ❞❡s❝r✐❜❡❞ ✐♥ ❆❧❣♦r✐t❤♠s ✹ ❛♥❞ ✺ ❝❛♥ ❜❡ ✉t✐❧✐③❡❞ ✐♥ ♥♦♥✲ ▼❛r❦♦✈✐❛♥ ❞♦♠❛✐♥s✳ ■♥ ❛❧❧ t❤r❡❡ ♠❡t❤♦❞s✱ ✇❡ ❝♦♥✈❡r❣❡ t♦ t❤❡ tr✉❡ ✈❛❧✉❡ ❢✉♥❝t✐♦♥✳ ▲❛st ❧❡❝t✉r❡✱ ✇❡ ♣r♦✈❡❞ t❤✐s r❡s✉❧t ❢♦r ❞②♥❛♠✐❝ ♣r♦❣r❛♠♠✐♥❣ ❜② ✉s✐♥❣ t❤❡ ❢❛❝t t❤❛t t❤❡ ❇❡❧❧♠❛♥ ❜❛❝❦✉♣ ♦♣❡r❛t♦r ✐s ❛ ❝♦♥tr❛❝t✐♦♥✳ ❲❡ s❛✇ ✐♥ t♦❞❛②✬s ❧❡❝t✉r❡ t❤❛t ▼♦♥t❡ ❈❛r❧♦ ♣♦❧✐❝② ❡✈❛❧✉❛t✐♦♥ ❝♦♥✈❡r❣❡s t♦ t❤❡ ♣♦❧✐❝②✬s ✈❛❧✉❡ ❢✉♥❝t✐♦♥ ❞✉❡ t♦ t❤❡ ❧❛✇ ♦❢ ❧❛r❣❡ ♥✉♠❜❡rs✳ ❚❉✭✵✮ ❝♦♥✈❡r❣❡s t♦ t❤❡ tr✉❡ ✈❛❧✉❡ ❛s ✇❡❧❧✱ ✇❤✐❝❤ ✇❡ ✇✐❧❧ ❧♦♦❦ ❛t ♠♦r❡ ❝❧♦s❡❧② ✐♥ t❤❡ ♥❡①t s❡❝t✐♦♥ ♦♥ ❜❛t❝❤ ❧❡❛r♥✐♥❣✳ ❇❡❝❛✉s❡ ✇❡ ❛r❡ t❛❦✐♥❣ ❛♥ ❛✈❡r❛❣❡ ♦✈❡r t❤❡ tr✉❡ ❞✐str✐❜✉t✐♦♥ ♦❢ r❡t✉r♥s ✐♥ ▼♦♥t❡ ❈❛r❧♦✱ ✇❡ ♦❜t❛✐♥ ❛♥ ✉♥❜✐❛s❡❞ ❡st✐♠❛t♦r ♦❢ t❤❡ ✈❛❧✉❡ ❛t ❡❛❝❤ st❛t❡✳ ❖♥ t❤❡ ♦t❤❡r ❤❛♥❞✱ ✐♥ ❚❉ ❧❡❛r♥✐♥❣✱ ✇❡ ❜♦♦tstr❛♣ t❤❡ ♥❡①t st❛t❡✬s ✈❛❧✉❡ ❡st✐♠❛t❡ t♦ ❣❡t t❤❡ ❝✉rr❡♥t st❛t❡✬s ✈❛❧✉❡ ❡st✐♠❛t❡✱ s♦ t❤❡ ❡st✐♠❛t❡ ✐s ❜✐❛s❡❞ ❜② t❤❡ ❡st✐♠❛t❡❞ ✈❛❧✉❡ ♦❢ t❤❡ ♥❡①t st❛t❡✳ ❋✉rt❤❡r ❞✐s❝✉ss✐♦♥s ♦♥ t❤✐s ❝❛♥ ❜❡ ❢♦✉♥❞ ✐♥ s❡❝t✐♦♥ ✻✳✷ ♦❢ ❙✉tt♦♥ ❛♥❞ ❇❛rt♦ ❬✶❪✳ ❚❤❡ ✈❛r✐❛♥❝❡ ♦❢ ▼♦♥t❡ ❈❛r❧♦ ❡✈❛❧✉❛t✐♦♥ ✐s r❡❧❛t✐✈❡❧② ❤✐❣❤❡r t❤❛♥ ❚❉ ❧❡❛r♥✐♥❣ ❜❡❝❛✉s❡ ✐♥ ▼♦♥t❡ ❈❛r❧♦ ❡✈❛❧✉❛t✐♦♥✱ ✇❡ ❝♦♥s✐❞❡r ♠❛♥② tr❛♥s✐t✐♦♥s ✐♥ ❡❛❝❤ ❡♣✐s♦❞❡ ✇✐t❤ ❡❛❝❤ tr❛♥s✐t✐♦♥ ❝♦♥tr✐❜✉t✐♥❣ ✈❛r✐❛♥❝❡ t♦ ♦✉r ❡st✐♠❛t❡✳ ❖♥ t❤❡ ♦t❤❡r ❤❛♥❞✱ ❚❉ ❧❡❛r♥✐♥❣ ♦♥❧② ❝♦♥s✐❞❡rs ♦♥❡ tr❛♥s✐t✐♦♥ ♣❡r ✉♣❞❛t❡✱ s♦ ✇❡ ❞♦ ♥♦t ❛❝❝✉♠✉❧❛t❡ ✈❛r✐❛♥❝❡ ❛s q✉✐❝❦❧②✳ ❋✐♥❛❧❧②✱ ▼♦♥t❡ ❈❛r❧♦ ✐s ❣❡♥❡r❛❧❧② ♠♦r❡ ❞❛t❛ ❡✣❝✐❡♥t t❤❛♥ ❚❉✭✵✮✳ ■♥ ▼♦♥t❡ ❈❛r❧♦✱ ✇❡ ✉♣❞❛t❡ t❤❡ ✈❛❧✉❡ ♦❢ ❛ st❛t❡ ❜❛s❡❞ ♦♥ t❤❡ r❡t✉r♥s ♦❢ t❤❡ ❡♥t✐r❡ ❡♣✐s♦❞❡✱ s♦ ✐❢ t❤❡r❡ ❛r❡ ❤✐❣❤❧② ♣♦s✐t✐✈❡ ♦r ♥❡❣❛t✐✈❡ r❡✇❛r❞s ♠❛♥② tr❛❥❡❝t♦r✐❡s ✐♥ t❤❡ ❢✉t✉r❡✱ t❤❡s❡ r❡✇❛r❞s ✇✐❧❧ st✐❧❧ ❜❡ ✐♠♠❡❞✐❛t❡❧② ✐♥❝♦r♣♦r❛t❡❞ ✐♥t♦ ♦✉r ✉♣❞❛t❡ ♦❢ t❤❡ ✈❛❧✉❡ ♦❢ t❤❡ st❛t❡✳ ❖♥ t❤❡ ♦t❤❡r ❤❛♥❞ ✐♥ ❚❉✭✵✮✱ ✇❡ ✉♣❞❛t❡ t❤❡ ✈❛❧✉❡ ♦❢ ❛ st❛t❡ ✉s✐♥❣ ♦♥❧② t❤❡ r❡✇❛r❞ ✐♥ t❤❡ ❝✉rr❡♥t st❡♣ ❛♥❞ s♦♠❡ ♣r❡✈✐♦✉s ❡st✐♠❛t❡ ♦❢ t❤❡ ✈❛❧✉❡ ❛t t❤❡ ♥❡①t st❛t❡✳ ❚❤✐s ♠❡❛♥s t❤❛t ✐❢ t❤❡r❡ ❛r❡ ❤✐❣❤❧② ♣♦s✐t✐✈❡ ♦r ♥❡❣❛t✐✈❡ r❡✇❛r❞s ♠❛♥② tr❛❥❡❝t♦r✐❡s ✐♥ t❤❡ ❢✉t✉r❡✱ ✇❡ ✇✐❧❧ ♦♥❧② ✐♥❝♦r♣♦r❛t❡ t❤❡s❡ ✐♥t♦ t❤❡ ❝✉rr❡♥t st❛t❡✬s ✈❛❧✉❡ ✉♣❞❛t❡ ✇❤❡♥ t❤❛t r❡✇❛r❞ ❤❛s ❜❡❡♥ ✉s❡❞ t♦ ✉♣❞❛t❡ t❤❡ ❜♦♦tstr❛♣ ❡st✐♠❛t❡ ♦❢ t❤❡ ♥❡①t st❛t❡✬s ✈❛❧✉❡✳ ❚❤✐s ♠❡❛♥s t❤❛t ✐❢ ❛ ❤✐❣❤❧② r❡✇❛r❞✐♥❣ ❡♣✐s♦❞❡ ❤❛s ❧❡♥❣t❤ L✱ t❤❡♥ ✇❡ ♠❛② ♥❡❡❞ t♦ ❡①♣❡r✐❡♥❝❡ t❤❛t ❡♣✐s♦❞❡ L t✐♠❡s ❢♦r t❤❡ ✐♥❢♦r♠❛t✐♦♥ ♦❢ t❤❡ ❤✐❣❤❧② r❡✇❛r❞✐♥❣ ❡♣✐s♦❞❡ t♦ tr❛✈❡❧ ❛❧❧ t❤❡ ✇❛② ❜❛❝❦ t♦ t❤❡ st❛rt✐♥❣ st❛t❡✳ ■♥ ❚❛❜❧❡ ✶✱ ✇❡ s✉♠♠❛r✐③❡ t❤❡ str❡♥❣t❤s ❛♥❞ ❧✐♠✐t❛t✐♦♥s ♦❢ ❡❛❝❤ ♠❡t❤♦❞ ❞✐s❝✉ss❡❞ ❤❡r❡✳ ✾

❉②♥❛♠✐❝ Pr♦❣r❛♠♠✐♥❣ ▼♦♥t❡ ❈❛r❧♦ ❚❡♠♣♦r❛❧ ❉✐✛❡r❡♥❝❡ ▼♦❞❡❧ ❋r❡❡❄ ◆♦ ❨❡s ❨❡s ◆♦♥✲❡♣✐s♦❞✐❝ ❞♦♠❛✐♥s❄ ❨❡s ◆♦ ❨❡s ◆♦♥✲▼❛r❦♦✈✐❛♥ ❞♦♠❛✐♥s❄ ◆♦ ❨❡s ◆♦ ❈♦♥✈❡r❣❡s t♦ tr✉❡ ✈❛❧✉❡ ❨❡s ❨❡s ❨❡s ❯♥❜✐❛s❡❞ ❊st✐♠❛t❡ ◆✴❆ ❨❡s ◆♦ ❱❛r✐❛♥❝❡ ◆✴❆ ❍✐❣❤ ▲♦✇ ❚❛❜❧❡ ✶✿ ❙✉♠♠❛r② ♦❢ ♠❡t❤♦❞s ✐♥ t❤✐s ❧❡❝t✉r❡✳ ❋✐❣✉r❡ ✺✿ ❊①❛♠♣❧❡ ✻✳✹ ❢r♦♠ ❙✉tt♦♥ ❛♥❞ ❇❛rt♦ ❬✶❪✳

✹✳✼ ❇❛t❝❤ ▼♦♥t❡ ❈❛r❧♦ ❛♥❞ ❚❡♠♣♦r❛❧ ❉✐✛❡r❡♥❝❡

❲❡ ♥♦✇ ❧♦♦❦ ❛t t❤❡ ❜❛t❝❤ ✈❡rs✐♦♥s ♦❢ t❤❡ ❛❧❣♦r✐t❤♠s ✐♥ t♦❞❛②✬s ❧❡❝t✉r❡✱ ✇❤❡r❡ ✇❡ ❤❛✈❡ ❛ s❡t ♦❢ ❤✐st♦r✐❡s t❤❛t ✇❡ ✉s❡ t♦ ♠❛❦❡ ✉♣❞❛t❡s ♠❛♥② t✐♠❡s✳ ❇❡❢♦r❡ ❧♦♦❦✐♥❣ ❛t t❤❡ ❜❛t❝❤ ❝❛s❡s ✐♥ ❣❡♥❡r❛❧✐t②✱ ❧❡t✬s ✜rst ❧♦♦❦ ❛t ❊①❛♠♣❧❡ ✻✳✹ ❢r♦♠ ❙✉tt♦♥ ❛♥❞ ❇❛rt♦ ❬✶❪ t♦ ♠♦r❡ ❝❧♦s❡❧② ❡①❛♠✐♥❡ t❤❡ ❞✐✛❡r❡♥❝❡ ❜❡t✇❡❡♥ ▼♦♥t❡ ❈❛r❧♦ ❛♥❞ ❚❉✭✵✮✳ ❙✉♣♣♦s❡ γ = 1 ❛♥❞ ✇❡ ❤❛✈❡ ❡✐❣❤t ❤✐st♦r✐❡s ❣❡♥❡r❛t❡❞ ❜② ♣♦❧✐❝② π✱ t❛❦❡ ❛❝t✐♦♥ act1 ✐♥ ❛❧❧ st❛t❡s✿ h1 = (A, act1, +0, B, act1, +0, terminal) hj = (B, act1, +1, terminal) ❢♦r j = 2, . . . , 7 h8 = (B, act1, +0, terminal). ❚❤❡♥✱ ✉s✐♥❣ ❡✐t❤❡r ❜❛t❝❤ ▼♦♥t❡ ❈❛r❧♦ ♦r ❚❉✭✵✮ ✇✐t❤ α = 1✱ ✇❡ s❡❡ t❤❛t V (B) = 0.75✳ ❍♦✇❡✈❡r✱ ✉s✐♥❣ ▼♦♥t❡ ❈❛r❧♦✱ ✇❡ ❣❡t t❤❛t V (A) = 0 s✐♥❝❡ ♦♥❧② t❤❡ ✜rst ❡♣✐s♦❞❡ ✈✐s✐ts st❛t❡ A ❛♥❞ ❤❛s r❡t✉r♥ ✵✳ ❖♥ t❤❡ ♦t❤❡r ❤❛♥❞✱ ❚❉✭✵✮ ❣✐✈❡s ✉s V (A) = 0.75 ❜❡❝❛✉s❡ ✇❡ ♣❡r❢♦r♠ t❤❡ ✉♣❞❛t❡ V (A) ← r1,1 + γV (B)✳ ❯♥❞❡r ❛ ▼❛r❦♦✈✐❛♥ ❞♦♠❛✐♥ ❧✐❦❡ t❤❡ ♦♥❡ s❤♦✇♥ ✐♥ ❋✐❣✉r❡ ✺✱ t❤❡ ❡st✐♠❛t❡ ❣✐✈❡♥ ❜② ❚❉✭✵✮ ♠❛❦❡s ♠♦r❡ s❡♥s❡✳ ■♥ t❤✐s s❡❝t✐♦♥✱ ✇❡ ❝♦♥s✐❞❡r t❤❡ ❜❛t❝❤ ❝❛s❡s ♦❢ ▼♦♥t❡ ❈❛r❧♦ ❛♥❞ ❚❉✭✵✮✳ ■♥ t❤❡ ❜❛t❝❤ ❝❛s❡✱ ✇❡ ❛r❡ ❣✐✈❡♥ ❛ ❜❛t❝❤✱ ♦r s❡t ♦❢ ❤✐st♦r✐❡s h1, . . . , hn✱ ✇❤✐❝❤ ✇❡ t❤❡♥ ❢❡❡❞ t❤r♦✉❣❤ ▼♦♥t❡ ❈❛r❧♦ ♦r ❚❉✭✵✮ ♠❛♥② t✐♠❡s✳ ❚❤❡ ♦♥❧② ❞✐✛❡r❡♥❝❡ ❢r♦♠ ♦✉r ❢♦r♠✉❧❛t✐♦♥s ❜❡❢♦r❡ ✐s t❤❛t ✇❡ ♦♥❧② ✉♣❞❛t❡ t❤❡ ✈❛❧✉❡ ❢✉♥❝t✐♦♥ ❛❢t❡r ❡❛❝❤ t✐♠❡ ✇❡ ♣r♦❝❡ss t❤❡ ❡♥t✐r❡ ❜❛t❝❤✳ ❚❤✉s ✐♥ ❚❉✭✵✮✱ t❤❡ ❜♦♦tstr❛♣ ❡st✐♠❛t❡ ✐s ✉♣❞❛t❡❞ ♦♥❧② ❛❢t❡r ❡❛❝❤ ♣❛ss t❤r♦✉❣❤ t❤❡ ❜❛t❝❤✳ ■♥ t❤❡ ▼♦♥t❡ ❈❛r❧♦ ❜❛t❝❤ s❡tt✐♥❣✱ t❤❡ ✈❛❧✉❡ ❛t ❡❛❝❤ st❛t❡ ❝♦♥✈❡r❣❡s t♦ t❤❡ ✈❛❧✉❡ t❤❛t ♠✐♥✐♠✐③❡s t❤❡ ♠❡❛♥ sq✉❛r❡❞ ❡rr♦r ✇✐t❤ t❤❡ ♦❜s❡r✈❡❞ r❡t✉r♥s✳ ❚❤✐s ❢♦❧❧♦✇s ❞✐r❡❝t❧② ❢r♦♠ t❤❡ ❢❛❝t t❤❛t ✐♥ ▼♦♥t❡ ❈❛r❧♦✱ ✇❡ t❛❦❡ ❛♥ ❛✈❡r❛❣❡ ♦✈❡r r❡t✉r♥s ❛t ❡❛❝❤ st❛t❡✱ ❛♥❞ ✐♥ ❣❡♥❡r❛❧✱ t❤❡ ▼❙❊ ♠✐♥✐♠✐③❡r ♦❢ s❛♠♣❧❡s ✐s ♣r❡❝✐s❡❧② t❤❡ ❛✈❡r❛❣❡ ♦❢ t❤❡ s❛♠♣❧❡s✳ ❚❤❛t ✐s✱ ❣✐✈❡♥ s❛♠♣❧❡s y1, . . . , yn✱ t❤❡ ✈❛❧✉❡ n

i=1(yi − ˆ

y)2 ✐s ♠✐♥✐♠✐③❡❞ ❢♦r ˆ y = n

i=1 yi✳ ❲❡ ❝❛♥ ❛❧s♦ s❡❡ t❤✐s ✐♥ t❤❡ ❡①❛♠♣❧❡ ❛t t❤❡ ❜❡❣✐♥♥✐♥❣ ♦❢ t❤❡ s❡❝t✐♦♥✳ ❲❡ ❣❡t t❤❛t

V (A) = 0 ❢♦r ▼♦♥t❡ ❈❛r❧♦ ❜❡❝❛✉s❡ t❤✐s ✐s t❤❡ ♦♥❧② ❤✐st♦r② ✈✐s✐t✐♥❣ st❛t❡ A✳ ■♥ t❤❡ ❚❉✭✵✮ ❜❛t❝❤ s❡tt✐♥❣✱ ✇❡ ❞♦ ♥♦t ❝♦♥✈❡r❣❡ t♦ t❤❡ s❛♠❡ r❡s✉❧t ❛s ✐♥ ▼♦♥t❡ ❈❛r❧♦✳ ■♥ t❤✐s ❝❛s❡✱ ✇❡ ✶✵

❝♦♥✈❡r❣❡ t♦ t❤❡ ✈❛❧✉❡ V π t❤❛t ✐s t❤❡ ✈❛❧✉❡ ♦❢ ♣♦❧✐❝② π ♦♥ t❤❡ ♠❛①✐♠✉♠ ❧✐❦❡❧✐❤♦♦❞ ▼❉P ♠♦❞❡❧ ✇❤❡r❡ ˆ P(s′|s, a) = 1 N(s, a)

n

• j=1

Lj−1

• t=1

✶(sj,t = s, aj,t, sj,t+1 = s′) ✭✷✷✮ ˆ r(s, a) = 1 N(s, a)

n

• j=1

Lj−1

• t=1

✶(sj,t = s, aj,t)rj,t. ✭✷✸✮ ■♥ ♦t❤❡r ✇♦r❞s✱ t❤❡ ♠❛①✐♠✉♠ ❧✐❦❡❧✐❤♦♦❞ ▼❉P ♠♦❞❡❧ ✐s t❤❡ ♠♦st ♥❛✐✈❡ ♠♦❞❡❧ ✇❡ ❝❛♥ ❝r❡❛t❡ ❜❛s❡❞ ♦♥ t❤❡ ❜❛t❝❤ ✲ t❤❡ tr❛♥s✐t✐♦♥ ♣r♦❜❛❜✐❧✐t② ˆ P(s′|s, a) ✐s t❤❡ ❢r❛❝t✐♦♥ ♦❢ t✐♠❡s t❤❛t ✇❡ s❡❡ t❤❡ tr❛♥s✐t✐♦♥ (s, a, s′) ❛❢t❡r ✇❡ t❛❦❡ ❛❝t✐♦♥ a ❛t st❛t❡ s ✐♥ t❤❡ ❜❛t❝❤✱ ❛♥❞ t❤❡ r❡✇❛r❞ ˆ r(s, a) ✐s t❤❡ ❛✈❡r❛❣❡ r❡✇❛r❞ ❡①♣❡r✐❡♥❝❡❞ ❛❢t❡r t❛❦✐♥❣ ❛❝t✐♦♥ a ❛t st❛t❡ s ✐♥ t❤❡ ❜❛t❝❤✳ ❲❡ ❛❧s♦ s❡❡ t❤✐s r❡s✉❧t ✐♥ t❤❡ ❡①❛♠♣❧❡ ❢r♦♠ t❤❡ ❜❡❣✐♥♥✐♥❣ ♦❢ t❤❡ s❡❝t✐♦♥✳ ■♥ t❤✐s ❝❛s❡✱ ♦✉r ♠❛①✐♠✉♠ ❧✐❦❡❧✐❤♦♦❞ ♠♦❞❡❧ ✐s ˆ P(B|A, act1) = 1 ✭✷✹✮ ˆ P(terminal|B, act1) = 1 ✭✷✺✮ ˆ r(A, act1) = 0 ✭✷✻✮ ˆ r(B, act1) = 0.75. ✭✷✼✮ ❚❤✐s ❣✐✈❡s ✉s V π(A) = 0.75✱ ❧✐❦❡ ✇❡ st❛t❡❞ ❜❡❢♦r❡✳ ❚❤❡ ✈❛❧✉❡ ❢✉♥❝t✐♦♥ ❞❡r✐✈❡❞ ❢r♦♠ t❤❡ ♠❛①✐♠✉♠ ❧✐❦❡❧✐❤♦♦❞ ▼❉P ♠♦❞❡❧ ✐s ❦♥♦✇♥ ❛s t❤❡ ❝❡rt❛✐♥t② ❡q✉✐✈❛❧❡♥❝❡ ❡st✐♠❛t❡✳ ❯s✐♥❣ t❤✐s r❡❧❛t✐♦♥s❤✐♣✱ ✇❡ ❤❛✈❡ ❛♥♦t❤❡r ♠❡t❤♦❞ ❢♦r ❡✈❛❧✉❛t✐♥❣ t❤❡ ♣♦❧✐❝②✳ ❲❡ ❝❛♥ ✜rst ❝♦♠♣✉t❡ t❤❡ ♠❛①✐♠✉♠ ❧✐❦❡❧✐❤♦♦❞ ▼❉P ♠♦❞❡❧ ✉s✐♥❣ t❤❡ ❜❛t❝❤✳ ❚❤❡♥ ✇❡ ❝❛♥ ❝♦♠♣✉t❡ V π ✉s✐♥❣ t❤✐s ♠♦❞❡❧ ❛♥❞ t❤❡ ♠♦❞❡❧✲❜❛s❡❞ ♣♦❧✐❝② ❡✈❛❧✉❛t✐♦♥ ♠❡t❤♦❞s ❞✐s❝✉ss❡❞ ✐♥ ❧❛st ❧❡❝t✉r❡✳ ❚❤✐s ♠❡t❤♦❞ ✐s ❤✐❣❤❧② ❞❛t❛ ❡✣❝✐❡♥t ❜✉t ✐s ❝♦♠♣✉t❛t✐♦♥❛❧❧② ❡①♣❡♥s✐✈❡ ❜❡❝❛✉s❡ ✐t ✐♥✈♦❧✈❡s s♦❧✈✐♥❣ t❤❡ ▼❉P ✇❤✐❝❤ t❛❦❡s t✐♠❡ O(|S|3) ❛♥❛❧②t✐❝❛❧❧② ❛♥❞ (|S|2|A|) ✈✐❛ ❞②♥❛♠✐❝ ♣r♦❣r❛♠♠✐♥❣✳ ❊①❡r❝✐s❡ ✹✳✸✳ ❈♦♥s✐❞❡r ❛❣❛✐♥ t❤❡ ▼❛rs ❘♦✈❡r ❡①❛♠♣❧❡ ✐♥ ❋✐❣✉r❡ ✸✳ ❙✉♣♣♦s❡ t❤❛t ♦✉r ❡st✐♠❛t❡ ❢♦r t❤❡ ✈❛❧✉❡ ♦❢ ❡❛❝❤ st❛t❡ ✐s ❝✉rr❡♥t❧② ✵✳ ■❢ ♦✉r ❜❛t❝❤ ❝♦♥s✐sts ♦❢ t✇♦ ❤✐st♦r✐❡s h1 = (S3, TL, +0, S2, TL, +0, S1, TL, +1, terminal) h2 = (S3, TL, +0, S2, TL, +0, S2, TL, +0, S1, TL, +1, terminal) ❛♥❞ ♦✉r ♣♦❧✐❝② ✐s TL✱ t❤❡♥ ✇❤❛t ✐s t❤❡ ❝❡rt❛✐♥t② ❡q✉✐✈❛❧❡♥❝❡ ❡st✐♠❛t❡❄

❘❡❢❡r❡♥❝❡s

❬✶❪ ❙✉tt♦♥✱ ❘✐❝❤❛r❞ ❙✳ ❛♥❞ ❆♥❞r❡✇ ●✳ ❇❛rt♦✳ ■♥tr♦❞✉❝t✐♦♥ t♦ ❘❡✐♥❢♦r❝❡♠❡♥t ▲❡❛r♥✐♥❣✳ ✷♥❞ ❡❞✳✱ ▼■❚ Pr❡ss✱ ✷✵✶✼✳ ❉r❛❢t✳ ❤tt♣✿✴✴✐♥❝♦♠♣❧❡t❡✐❞❡❛s✳♥❡t✴❜♦♦❦✴t❤❡✲❜♦♦❦✲✷♥❞✳❤t♠❧✳ ✶✶