A Partition-Based First-Order Probabilistic Logic to Represent - - PowerPoint PPT Presentation

A Partition-Based First-Order Probabilistic Logic to Represent Interactive Beliefs

Alessandro Panella and Piotr Gmytrasiewicz

Fifth International Conference on Scalable Uncertainty Management

Dayton, OH October 10, 2011

Panella and Gmytrasiewicz (CS Dept. - UIC) A FOPL for Interactive Beliefs October 10, 2011 1 / 18

Outline

1

Quick Look

2

Introduction The Problem Related Work

3

Proposed Formalization 0-th Level Beliefs 1st Level Beliefs n-th Level Beliefs

4

Conclusion

5

Bibliography

Panella and Gmytrasiewicz (CS Dept. - UIC) A FOPL for Interactive Beliefs October 10, 2011 2 / 18

Quick Look

Contribution

Formalization of a theoretical framework that allows to compactly represent interactive beliefs

Probability theory (First-Order) Logic Maximum Entropy

Main idea: recursive partitioning of the belief simplices

(b) (c) (d) (a)

...

Panella and Gmytrasiewicz (CS Dept. - UIC) A FOPL for Interactive Beliefs October 10, 2011 3 / 18

Introduction The Problem

Stochastic Planning

The need for compact representations: Use of first-order logic: Describe sets of states Capture regularities

Panella and Gmytrasiewicz (CS Dept. - UIC) A FOPL for Interactive Beliefs October 10, 2011 4 / 18

Introduction The Problem

Stochastic Planning

The need for compact representations: Use of first-order logic: Describe sets of states Capture regularities

Panella and Gmytrasiewicz (CS Dept. - UIC) A FOPL for Interactive Beliefs October 10, 2011 4 / 18

Introduction The Problem

Stochastic Planning

An Example

n × n grid world Actions: UP , DOWN, LEFT, RIGHT Probabilistic transition function:

For every location, P(succeed) = .9

GO RIGHT GO DOWN Panella and Gmytrasiewicz (CS Dept. - UIC) A FOPL for Interactive Beliefs October 10, 2011 5 / 18

Introduction The Problem

Stochastic Planning

An Example

n × n grid world Actions: UP , DOWN, LEFT, RIGHT Probabilistic transition function:

For every location, P(succeed) = .9

GO RIGHT GO DOWN Panella and Gmytrasiewicz (CS Dept. - UIC) A FOPL for Interactive Beliefs October 10, 2011 5 / 18

Introduction The Problem

Stochastic Planning

An Example

n × n grid world Actions: UP , DOWN, LEFT, RIGHT Probabilistic transition function:

For every location, P(succeed) = .9

GO RIGHT GO DOWN Panella and Gmytrasiewicz (CS Dept. - UIC) A FOPL for Interactive Beliefs October 10, 2011 5 / 18

Introduction The Problem

Interactive Settings

Representation needs even more stringent

Panella and Gmytrasiewicz (CS Dept. - UIC) A FOPL for Interactive Beliefs October 10, 2011 6 / 18

Introduction The Problem

Interactive Settings

Representation needs even more stringent

Panella and Gmytrasiewicz (CS Dept. - UIC) A FOPL for Interactive Beliefs October 10, 2011 6 / 18

Introduction The Problem

Interactive Settings

Representation needs even more stringent

Panella and Gmytrasiewicz (CS Dept. - UIC) A FOPL for Interactive Beliefs October 10, 2011 6 / 18

Introduction The Problem

Interactive Settings

Representation needs even more stringent

Panella and Gmytrasiewicz (CS Dept. - UIC) A FOPL for Interactive Beliefs October 10, 2011 6 / 18

Introduction The Problem

(Finitely Nested) Interactive POMDPs

Level-n belief bi,n ∈ ∆(ISi,n), where ISi,0 = S ISi,1 = S × ∆(ISj,0) . . . ISi,n = S × ∆(ISj,n−1) Value function of (I-)POMDPs is Piecewise linear and convex. Divides the simplex into behavior-equivalent partitions.

(0, 1) (1, 0) (0, 0)

s1 s2

From Kaelbling et al. (1998) Panella and Gmytrasiewicz (CS Dept. - UIC) A FOPL for Interactive Beliefs October 10, 2011 7 / 18

Introduction The Problem

(Finitely Nested) Interactive POMDPs

Level-n belief bi,n ∈ ∆(ISi,n), where ISi,0 = S ISi,1 = S × ∆(ISj,0) . . . ISi,n = S × ∆(ISj,n−1) Value function of (I-)POMDPs is Piecewise linear and convex. Divides the simplex into behavior-equivalent partitions.

(0, 1) (1, 0) (0, 0)

s1 s2

From Kaelbling et al. (1998) Panella and Gmytrasiewicz (CS Dept. - UIC) A FOPL for Interactive Beliefs October 10, 2011 7 / 18

Introduction Related Work

Related Work

First Order Probabilistic Languages: Seminal theoretical work (Nilsson, 1986; Halpern, 1989) Recent practical approaches:

BLOG (Milch et al., 2005), Markov Logic (Richardson and Domingos, 2006), . . .

Relational stochastic planning: Relational MDPs (Boutilier et al., 2001; Sanner and Boutilier, 2009) Relational POMDPs (Sanner and Kersting, 2010; Wang and Khardon, 2010) Belief hierarchies Extensive treatment in Game Theory, starting from Bayesian Games (Harsanyi, 1967; Aumann, 1999) Probabilistic modal logics (Fagin and Halpern, 1994; Shirazi and Amir, 2008) Interactive POMDPs (Gmytrasiewicz and Doshi, 2005)

Panella and Gmytrasiewicz (CS Dept. - UIC) A FOPL for Interactive Beliefs October 10, 2011 8 / 18

Introduction Related Work

Related Work

First Order Probabilistic Languages: Seminal theoretical work (Nilsson, 1986; Halpern, 1989) Recent practical approaches:

BLOG (Milch et al., 2005), Markov Logic (Richardson and Domingos, 2006), . . .

Relational stochastic planning: Relational MDPs (Boutilier et al., 2001; Sanner and Boutilier, 2009) Relational POMDPs (Sanner and Kersting, 2010; Wang and Khardon, 2010) Belief hierarchies Extensive treatment in Game Theory, starting from Bayesian Games (Harsanyi, 1967; Aumann, 1999) Probabilistic modal logics (Fagin and Halpern, 1994; Shirazi and Amir, 2008) Interactive POMDPs (Gmytrasiewicz and Doshi, 2005)

Panella and Gmytrasiewicz (CS Dept. - UIC) A FOPL for Interactive Beliefs October 10, 2011 8 / 18

Introduction Related Work

Related Work

First Order Probabilistic Languages: Seminal theoretical work (Nilsson, 1986; Halpern, 1989) Recent practical approaches:

BLOG (Milch et al., 2005), Markov Logic (Richardson and Domingos, 2006), . . .

Relational stochastic planning: Relational MDPs (Boutilier et al., 2001; Sanner and Boutilier, 2009) Relational POMDPs (Sanner and Kersting, 2010; Wang and Khardon, 2010) Belief hierarchies Extensive treatment in Game Theory, starting from Bayesian Games (Harsanyi, 1967; Aumann, 1999) Probabilistic modal logics (Fagin and Halpern, 1994; Shirazi and Amir, 2008) Interactive POMDPs (Gmytrasiewicz and Doshi, 2005)

Panella and Gmytrasiewicz (CS Dept. - UIC) A FOPL for Interactive Beliefs October 10, 2011 8 / 18

Proposed Formalization 0-th Level Beliefs

Grid World Example

n × n grid Agent i tagging a moving target j

1 2 3 4 5 6 1 2 3 4 5 6

Uncertainty about target’s position: predicate jPos(x, y) Auxiliary deterministic predicates:

geq(x, k) ≡ x ≥ k leq(x, k) ≡ x ≤ k

Panella and Gmytrasiewicz (CS Dept. - UIC) A FOPL for Interactive Beliefs October 10, 2011 9 / 18

Proposed Formalization 0-th Level Beliefs

Level-0 Belief Base

i’s belief about the state of the world

Bi,0 = φ1, α1 : : φm, αm

φ0 φ1 S ψ0 ψ1 ψ2

φk’s are arbitrary sentences in predicate logic, and αk ∈ [0, 1]; ψ’s are the induced partitions – (Ψ, 2Ψ, pi,0) Only partial specification of distribution To obtain unique distribution: Maximum Entropy (max-ent): max

pi,0

• −
• ψ∈ΨB

pi,0(S(ψ)) log pi,0(S(ψ))

• Panella and Gmytrasiewicz (CS Dept. - UIC)

A FOPL for Interactive Beliefs October 10, 2011 10 / 18

Proposed Formalization 0-th Level Beliefs

Grid World Example (cont’d)

Assume agent interested in horizontal position of target w.r.t. center: Bi,0 =

∃x, y(jPos(x, y) ∧ leq(x, ⌊n/2⌋)), 0.8 ∃x, y(jPos(x, y) ∧ geq(x, ⌊n/2⌋)), 0.5

φ0 φ1 S

ψ0 ψ1 ψ2

In this case, unique distribution pi,0 = (0.5, 0.3, 0.2).

Panella and Gmytrasiewicz (CS Dept. - UIC) A FOPL for Interactive Beliefs October 10, 2011 11 / 18

Proposed Formalization 1st Level Beliefs

Level-1 Belief Base

i’s belief about j’s belief

Bi,1 = φj,0

1 ,

α1 : : φj,0

m ,

αm (1)

φj,0

k is of the form Pj(φ) △ β,

△∈ {<, ≤, =, ≥, >}, β ∈ [0, 1]; The sentences φj,0

k induce a partitioning on j’s L-0 belief simplex;

S ΦB ΨB ∆(ΨB) Φj,0

B

Ψj,0

B ψ ∈ Ψj,0

B

pi,1

i

(a) (b) (c)

Panella and Gmytrasiewicz (CS Dept. - UIC) A FOPL for Interactive Beliefs October 10, 2011 12 / 18

Proposed Formalization 1st Level Beliefs

Grid World Example (cont’d)

Target j models agent i’s beliefs about j’s position Bj,1 =

Pi(φ0) ≥ 0.4, 0.4 Pi(φ1) > 0.5, 0.7

ψ0 ψ1 ψ2

pi(φ1) = 0.5 pi(φ0) = 0.4

φ0 φ1 S

ψi,0

2

ψi,0

1

ψi,0

State of the world: i's L0 simplex:

ψ0 ψ1 ψ2 φi,0 φi,0

1

Unique consistent distribution pj,1 = (0.3, 0.1, 0.6).

Panella and Gmytrasiewicz (CS Dept. - UIC) A FOPL for Interactive Beliefs October 10, 2011 13 / 18

Proposed Formalization n-th Level Beliefs

Level-n Belief Base

Generalize the procedure to an arbitrary nesting level: i’s beliefs about j’s beliefs about i’s beliefs about. . . Bn

i =

φj,n−1

1

, α1 : : φj,n−1

m

αm

φj,n−1

k

is of the form Pj(φi,n−2) △ β; The core idea is that the partitions at level l − 1 constitute the vertices

• f the simplex at level l

(b) (c) (d) (a)

...

Panella and Gmytrasiewicz (CS Dept. - UIC) A FOPL for Interactive Beliefs October 10, 2011 14 / 18

Proposed Formalization n-th Level Beliefs

Level-n Belief Base

Generalize the procedure to an arbitrary nesting level: i’s beliefs about j’s beliefs about i’s beliefs about. . . Bn

i =

φj,n−1

1

, α1 : : φj,n−1

m

αm

φj,n−1

k

is of the form Pj(φi,n−2) △ β; The core idea is that the partitions at level l − 1 constitute the vertices

• f the simplex at level l

(b) (c) (d) (a)

...

Panella and Gmytrasiewicz (CS Dept. - UIC) A FOPL for Interactive Beliefs October 10, 2011 14 / 18

Proposed Formalization n-th Level Beliefs

Grid World Example (cont’d)

Agent i models target j’s beliefs about i’s beliefs about j’s position Bi,2 =

Pj(Pi(φ0) ≥ 0.4) < 0.4, 0.2 Pj(Pi(φ1) > 0.5) < 0.7, 0.6

ψi,0 ψi,0

1

ψi,1

2 pi(φi,0

0 ) = 0.7

pi(φi,0

0 ) = 0.6

ψ0 ψ1 ψ2

pi(φ1) = 0.5 pi(φ0) = 0.4

φ0 φ1 S

ψi,0

2

ψi,0

1

ψi,0

ψj,1 ψj,1

1

ψj,1

2

ψj,1

3 State of the world: i's L0 simplex: j's L1 simplex:

ψ0 ψ1 ψ2 φi,0 φi,0

1

φj,1 φj,1

1

Max-ent optimization gives pi,2 = (0.48, 0.32, 0.08, 0.12).

Panella and Gmytrasiewicz (CS Dept. - UIC) A FOPL for Interactive Beliefs October 10, 2011 15 / 18

Conclusion

Conclusion and Future Work

Contribution: Formalization of a FOPL that allows to represent interactive beliefs through recursively partitioning the belief simplices. Future Work: Practical implementation of (subset of) the proposed formalism;

Extension of existing approaches to interactive beliefs semantics; E.g. BLOG, Markov Logic, Relational Probabilistic Models, . . .

Embed the formalism in a (multi-agent) stochastic planning algorithm

Piecewise linearity of value functions in (I-)POMDPs makes the proposed approach promising; Extend approaches like Sanner et al., 2010 (Sanner and Kersting, 2010).

Panella and Gmytrasiewicz (CS Dept. - UIC) A FOPL for Interactive Beliefs October 10, 2011 16 / 18

Conclusion

Conclusion and Future Work

Contribution: Formalization of a FOPL that allows to represent interactive beliefs through recursively partitioning the belief simplices. Future Work: Practical implementation of (subset of) the proposed formalism;

Extension of existing approaches to interactive beliefs semantics; E.g. BLOG, Markov Logic, Relational Probabilistic Models, . . .

Embed the formalism in a (multi-agent) stochastic planning algorithm

Piecewise linearity of value functions in (I-)POMDPs makes the proposed approach promising; Extend approaches like Sanner et al., 2010 (Sanner and Kersting, 2010).

Panella and Gmytrasiewicz (CS Dept. - UIC) A FOPL for Interactive Beliefs October 10, 2011 16 / 18

Conclusion

The end. Any questions? Thank you!

Panella and Gmytrasiewicz (CS Dept. - UIC) A FOPL for Interactive Beliefs October 10, 2011 17 / 18

Bibliography

Bibliography

• R. J. Aumann. Interactive epistemology II: Probability. International Journal of Game Theory, 28(3):301–314, 1999.
• C. Boutilier, R. Reiter, and B. Price. Symbolic Dynamic Programming for First-Order MDPs. In IJCAI 01, pages 690–700. Morgan

Kaufmann, 2001.

• R. Fagin and J. Y. Halpern. Reasoning about knowledge and probability. J. ACM, 41:340–367, March 1994. ISSN 0004-5411.

P . J. Gmytrasiewicz and P . Doshi. A framework for sequential planning in multi-agent settings. Journal of Artificial Intelligence Research, 24:24–49, 2005.

• J. Y. Halpern. An analysis of first-order logics of probability. In Proceedings of the 11th international joint conference on Artificial

intelligence - Volume 2, pages 1375–1381, San Francisco, CA, USA, 1989. Morgan Kaufmann Publishers Inc.

• J. C. Harsanyi. Games with Incomplete Information Played by “Bayesian” Players, I-III. Part I. The Basic Model. Management

Science, 14(3):159–182, 1967. doi: 10.2307/2628393.

• L. P

. Kaelbling, M. L. Littman, and A. R. Cassandra. Planning and acting in partially observable stochastic domains. Artificial Intelligence, 101:99–134, 1998.

• B. Milch, B. Marthi, S. J. Russell, D. Sontag, D. L. Ong, and A. Kolobov. Blog: Probabilistic models with unknown objects. In

Probabilistic, Logical and Relational Learning, 2005.

• N. J. Nilsson. Probabilistic logic. Artif. Intell., 28:71–88, February 1986. ISSN 0004-3702. doi: 10.1016/0004-3702(86)90031-7.
• M. Richardson and P

. Domingos. Markov logic networks. Mach. Learn., 62:107–136, February 2006. ISSN 0885-6125. doi: 10.1007/s10994-006-5833-1.

• S. Sanner and C. Boutilier. Practical solution techniques for first-order MDPs. Artif. Intell., 173:748–788, April 2009. ISSN

0004-3702. doi: 10.1016/j.artint.2008.11.003.

• S. Sanner and K. Kersting. Symbolic dynamic programming for first-order POMDPs. In AAAI 10, pages 1140–1146, 2010.
• A. Shirazi and E. Amir. Factored models for probabilistic modal logic. In Proceedings of the 23rd national conference on Artificial

intelligence - Volume 1, pages 541–547. AAAI Press, 2008. ISBN 978-1-57735-368-3.

• C. Wang and R. Khardon. Relational partially observable MDPs. In AAAI 10, pages 1153–1157, 2010.

Panella and Gmytrasiewicz (CS Dept. - UIC) A FOPL for Interactive Beliefs October 10, 2011 18 / 18