GPU-accelerated End-to-end Differentiable Planning and Reasoning - - PowerPoint PPT Presentation

GPU-accelerated End-to-end Differentiable Planning and Reasoning

Tim Rockt¨ aschel

Whiteson Research Lab, University of Oxford http://rockt.github.com Twitter: @ rockt tim.rocktaschel@cs.ox.ac.uk

Talk at GTC Europe, ID 23372 12th of October 2017

XKCD, 17th May 2017

Tim Rockt¨ aschel GPU-accelerated End-to-end Differentiable Planning and Reasoning 2/39

XKCD, 17th May 2017

Data & Explanations

• Rules
• (Partial) Programs
• Natural Language

Tim Rockt¨ aschel GPU-accelerated End-to-end Differentiable Planning and Reasoning 2/39

XKCD, 17th May 2017

Data & Explanations

• Rules
• (Partial) Programs
• Natural Language

Answers & Explanations

• Rules
• Programs
• Natural Language
• Plans

Tim Rockt¨ aschel GPU-accelerated End-to-end Differentiable Planning and Reasoning 2/39

XKCD, 17th May 2017

Data & Explanations

• Rules
• (Partial) Programs
• Natural Language

Data & Explanations

• Rules
• (Partial) Programs
• Natural Language

Answers & Explanations

• Rules
• Programs
• Natural Language
• Plans

Answers & Explanations

• Rules
• Programs
• Natural Language
• Plans

Tim Rockt¨ aschel GPU-accelerated End-to-end Differentiable Planning and Reasoning 2/39

XKCD, 17th May 2017

Data & Explanations

• Rules
• (Partial) Programs
• Natural Language

Data & Explanations

• Rules
• (Partial) Programs
• Natural Language

Answers & Explanations

• Rules
• Programs
• Natural Language
• Plans

Answers & Explanations

• Rules
• Programs
• Natural Language
• Plans

Data Efficiency & Model Interpretability

Tim Rockt¨ aschel GPU-accelerated End-to-end Differentiable Planning and Reasoning 2/39

Joint work with

• 1. End-to-end Differentiable Reasoning, Application: Knowledge Base Inference, NIPS 2017

Sebastian Riedel, University College London Tim Rockt¨ aschel GPU-accelerated End-to-end Differentiable Planning and Reasoning 3/39

Joint work with

• 1. End-to-end Differentiable Reasoning, Application: Knowledge Base Inference, NIPS 2017

Sebastian Riedel, University College London

• 2. End-to-end Differentiable Planning, Application: Atari, work-in-progress

Gregory Farquhar Maximilian Igl Shimon Whiteson University of Oxford Tim Rockt¨ aschel GPU-accelerated End-to-end Differentiable Planning and Reasoning 3/39

End-to-end Differentiable Reasoning

Expert Systems

• No/little training data
• Interpretable

Expert Systems

• No/little training data
• Interpretable
• Behavior manually defined
• No generalization

Expert Systems

• No/little training data
• Interpretable
• Behavior manually defined
• No generalization

Expert Systems

• No/little training data
• Interpretable
• Behavior manually defined
• No generalization

Representation Learning

• Behavior learned
• Strong generalization

Expert Systems

• No/little training data
• Interpretable
• Behavior manually defined
• No generalization

Representation Learning

• Behavior learned
• Strong generalization
• Lot of training data needed
• Not interpretable

Expert Systems

• No/little training data
• Interpretable

Representation Learning

• Behavior learned
• Strong generalization

Aims

Modular construction of neural networks for end-to-end differentiable reasoning in knowledge bases

Tim Rockt¨ aschel GPU-accelerated End-to-end Differentiable Planning and Reasoning 7/39

Aims

Modular construction of neural networks for end-to-end differentiable reasoning in knowledge bases Incorporate background knowledge in form of rules ⇒ Data Efficiency

Tim Rockt¨ aschel GPU-accelerated End-to-end Differentiable Planning and Reasoning 7/39

Aims

Modular construction of neural networks for end-to-end differentiable reasoning in knowledge bases Incorporate background knowledge in form of rules ⇒ Data Efficiency Calculate gradient of proof success w.r.t. subsymbolic representations

Tim Rockt¨ aschel GPU-accelerated End-to-end Differentiable Planning and Reasoning 7/39

Aims

Modular construction of neural networks for end-to-end differentiable reasoning in knowledge bases Incorporate background knowledge in form of rules ⇒ Data Efficiency Calculate gradient of proof success w.r.t. subsymbolic representations Rule application is explicit, but symbol comparison is neural

Tim Rockt¨ aschel GPU-accelerated End-to-end Differentiable Planning and Reasoning 7/39

Aims

Modular construction of neural networks for end-to-end differentiable reasoning in knowledge bases Incorporate background knowledge in form of rules ⇒ Data Efficiency Calculate gradient of proof success w.r.t. subsymbolic representations Rule application is explicit, but symbol comparison is neural Use similarity between vector representations of symbols in proofs

Tim Rockt¨ aschel GPU-accelerated End-to-end Differentiable Planning and Reasoning 7/39

Aims

Modular construction of neural networks for end-to-end differentiable reasoning in knowledge bases Incorporate background knowledge in form of rules ⇒ Data Efficiency Calculate gradient of proof success w.r.t. subsymbolic representations Rule application is explicit, but symbol comparison is neural Use similarity between vector representations of symbols in proofs Learn vector representations of symbols from data using gradient descent

Tim Rockt¨ aschel GPU-accelerated End-to-end Differentiable Planning and Reasoning 7/39

Aims

Modular construction of neural networks for end-to-end differentiable reasoning in knowledge bases Incorporate background knowledge in form of rules ⇒ Data Efficiency Calculate gradient of proof success w.r.t. subsymbolic representations Rule application is explicit, but symbol comparison is neural Use similarity between vector representations of symbols in proofs Learn vector representations of symbols from data using gradient descent Induce interpretable logical rules from data by gradient descent ⇒ Model Interpretability

Tim Rockt¨ aschel GPU-accelerated End-to-end Differentiable Planning and Reasoning 7/39

Task: Link Prediction

Real world knowledge bases (like Freebase, DBPedia, YAGO, etc.) are incomplete!

Das et al. (2017) 8/39

Task: Link Prediction

Real world knowledge bases (like Freebase, DBPedia, YAGO, etc.) are incomplete! placeOfBirth attribute is missing for 71% of people!

Das et al. (2017) 8/39

Task: Link Prediction

Real world knowledge bases (like Freebase, DBPedia, YAGO, etc.) are incomplete! placeOfBirth attribute is missing for 71% of people! Commonsense knowledge often not stated explicitly

Das et al. (2017) 8/39

Task: Link Prediction

Real world knowledge bases (like Freebase, DBPedia, YAGO, etc.) are incomplete! placeOfBirth attribute is missing for 71% of people! Commonsense knowledge often not stated explicitly Weak logical relationships that can be used for inferring facts

Das et al. (2017) 8/39

Task: Link Prediction

Real world knowledge bases (like Freebase, DBPedia, YAGO, etc.) are incomplete! placeOfBirth attribute is missing for 71% of people! Commonsense knowledge often not stated explicitly Weak logical relationships that can be used for inferring facts melinda seattle livesIn?

Das et al. (2017) 8/39

Task: Link Prediction

Real world knowledge bases (like Freebase, DBPedia, YAGO, etc.) are incomplete! placeOfBirth attribute is missing for 71% of people! Commonsense knowledge often not stated explicitly Weak logical relationships that can be used for inferring facts melinda bill spouseOf seattle livesIn?

Das et al. (2017) 8/39

Task: Link Prediction

Real world knowledge bases (like Freebase, DBPedia, YAGO, etc.) are incomplete! placeOfBirth attribute is missing for 71% of people! Commonsense knowledge often not stated explicitly Weak logical relationships that can be used for inferring facts melinda bill spouseOf microsoft chairmanOf seattle livesIn?

Das et al. (2017) 8/39

Task: Link Prediction

Real world knowledge bases (like Freebase, DBPedia, YAGO, etc.) are incomplete! placeOfBirth attribute is missing for 71% of people! Commonsense knowledge often not stated explicitly Weak logical relationships that can be used for inferring facts melinda bill spouseOf microsoft chairmanOf seattle headquarteredIn livesIn?

Das et al. (2017) 8/39

Notation

Constant: homer, bart, lisa etc. (lowercase)

Tim Rockt¨ aschel GPU-accelerated End-to-end Differentiable Planning and Reasoning 9/39

Notation

Constant: homer, bart, lisa etc. (lowercase) Variable: X, Y etc. (uppercase, universally quantified)

Tim Rockt¨ aschel GPU-accelerated End-to-end Differentiable Planning and Reasoning 9/39

Notation

Constant: homer, bart, lisa etc. (lowercase) Variable: X, Y etc. (uppercase, universally quantified) Term: constant or variable

Tim Rockt¨ aschel GPU-accelerated End-to-end Differentiable Planning and Reasoning 9/39

Notation

Constant: homer, bart, lisa etc. (lowercase) Variable: X, Y etc. (uppercase, universally quantified) Term: constant or variable Predicate: fatherOf, parentOf etc. function from terms to a Boolean

Tim Rockt¨ aschel GPU-accelerated End-to-end Differentiable Planning and Reasoning 9/39

Notation

Constant: homer, bart, lisa etc. (lowercase) Variable: X, Y etc. (uppercase, universally quantified) Term: constant or variable Predicate: fatherOf, parentOf etc. function from terms to a Boolean Atom: predicate and terms, e.g., parentOf(X, bart)

Tim Rockt¨ aschel GPU-accelerated End-to-end Differentiable Planning and Reasoning 9/39

Notation

Constant: homer, bart, lisa etc. (lowercase) Variable: X, Y etc. (uppercase, universally quantified) Term: constant or variable Predicate: fatherOf, parentOf etc. function from terms to a Boolean Atom: predicate and terms, e.g., parentOf(X, bart) Rule: head :– body. head: atom body: (possibly empty) list of atoms representing conjunction grandfatherOf(X, Y) :– fatherOf(X, Z), parentOf(Z, Y).

Tim Rockt¨ aschel GPU-accelerated End-to-end Differentiable Planning and Reasoning 9/39

Notation

Constant: homer, bart, lisa etc. (lowercase) Variable: X, Y etc. (uppercase, universally quantified) Term: constant or variable Predicate: fatherOf, parentOf etc. function from terms to a Boolean Atom: predicate and terms, e.g., parentOf(X, bart) Rule: head :– body. head: atom body: (possibly empty) list of atoms representing conjunction grandfatherOf(X, Y) :– fatherOf(X, Z), parentOf(Z, Y). Fact: ground rule (no free variables) with empty body, e.g., parentOf(homer, bart).

Tim Rockt¨ aschel GPU-accelerated End-to-end Differentiable Planning and Reasoning 9/39

Example Knowledge Base

1. fatherOf(abe, homer). 2. parentOf(homer, lisa). 3. parentOf(homer, bart). 4. grandpaOf(abe, lisa). 5. grandfatherOf(abe, maggie).

Tim Rockt¨ aschel GPU-accelerated End-to-end Differentiable Planning and Reasoning 10/39

Example Knowledge Base

1. fatherOf(abe, homer). 2. parentOf(homer, lisa). 3. parentOf(homer, bart). 4. grandpaOf(abe, lisa). 5. grandfatherOf(abe, maggie). 6. grandfatherOf(X1, Y1) :– fatherOf(X1, Z1), parentOf(Z1, Y1). 7. grandparentOf(X2, Y2) :– grandfatherOf(X2, Y2).

Tim Rockt¨ aschel GPU-accelerated End-to-end Differentiable Planning and Reasoning 10/39

Prolog Backward Chaining Example

Example Knowledge Base:

• 1. fatherOf(abe, homer).
• 2. parentOf(homer, bart).
• 3. grandfatherOf(X, Y) :–

fatherOf(X, Z), parentOf(Z, Y).

Tim Rockt¨ aschel GPU-accelerated End-to-end Differentiable Planning and Reasoning 11/39

Prolog Backward Chaining Example

Example Knowledge Base:

• 1. fatherOf(abe, homer).
• 2. parentOf(homer, bart).
• 3. grandfatherOf(X, Y) :–

fatherOf(X, Z), parentOf(Z, Y). grandfatherOf(abe, bart)?

Tim Rockt¨ aschel GPU-accelerated End-to-end Differentiable Planning and Reasoning 11/39

Prolog Backward Chaining Example

Example Knowledge Base:

• 1. fatherOf(abe, homer).
• 2. parentOf(homer, bart).
• 3. grandfatherOf(X, Y) :–

fatherOf(X, Z), parentOf(Z, Y). grandfatherOf(abe, bart)? failure failure success {X/abe, Y/bart} 3.1 fatherOf(abe, Z)? 1 2 3

Tim Rockt¨ aschel GPU-accelerated End-to-end Differentiable Planning and Reasoning 11/39

Prolog Backward Chaining Example

Example Knowledge Base:

• 1. fatherOf(abe, homer).
• 2. parentOf(homer, bart).
• 3. grandfatherOf(X, Y) :–

fatherOf(X, Z), parentOf(Z, Y). grandfatherOf(abe, bart)? failure failure success {X/abe, Y/bart} 3.1 fatherOf(abe, Z)? 1 2 3 success {X/abe, Y/bart, Z/homer} 3.2 parentOf(homer, bart)? failure failure 1 2 3

Tim Rockt¨ aschel GPU-accelerated End-to-end Differentiable Planning and Reasoning 11/39

Prolog Backward Chaining Example

Example Knowledge Base:

• 1. fatherOf(abe, homer).
• 2. parentOf(homer, bart).
• 3. grandfatherOf(X, Y) :–

fatherOf(X, Z), parentOf(Z, Y). grandfatherOf(abe, bart)? failure failure success {X/abe, Y/bart} 3.1 fatherOf(abe, Z)? 1 2 3 success {X/abe, Y/bart, Z/homer} 3.2 parentOf(homer, bart)? failure failure 1 2 3 failure success {X/abe, Y/bart, Z/homer} failure 1 2 3

Tim Rockt¨ aschel GPU-accelerated End-to-end Differentiable Planning and Reasoning 11/39

Prolog Backward Chaining Example

Example Knowledge Base:

• 1. fatherOf(abe, homer).
• 2. parentOf(homer, bart).
• 3. grandfatherOf(X, Y) :–

fatherOf(X, Z), parentOf(Z, Y). grandfatherOf(abe, bart)? failure failure success {X/abe, Y/bart} 3.1 fatherOf(abe, Z)? 1 2 3 success {X/abe, Y/bart, Z/homer} 3.2 parentOf(homer, bart)? failure failure 1 2 3 failure success {X/abe, Y/bart, Z/homer} failure 1 2 3

What about grandpaOf(abe, bart)?

Tim Rockt¨ aschel GPU-accelerated End-to-end Differentiable Planning and Reasoning 11/39

Symbolic Representations

Symbols (constants and predicates) do not share any information: grandpaOf = grandfatherOf

Tim Rockt¨ aschel GPU-accelerated End-to-end Differentiable Planning and Reasoning 12/39

Symbolic Representations

Symbols (constants and predicates) do not share any information: grandpaOf = grandfatherOf No notion of similarity: apple ∼ orange, professorAt ∼ lecturerAt

Tim Rockt¨ aschel GPU-accelerated End-to-end Differentiable Planning and Reasoning 12/39

Symbolic Representations

Symbols (constants and predicates) do not share any information: grandpaOf = grandfatherOf No notion of similarity: apple ∼ orange, professorAt ∼ lecturerAt No generalization beyond what can be symbolically inferred: isFruit(apple), apple ∼ organge, isFruit(orange)?

Tim Rockt¨ aschel GPU-accelerated End-to-end Differentiable Planning and Reasoning 12/39

Symbolic Representations

Symbols (constants and predicates) do not share any information: grandpaOf = grandfatherOf No notion of similarity: apple ∼ orange, professorAt ∼ lecturerAt No generalization beyond what can be symbolically inferred: isFruit(apple), apple ∼ organge, isFruit(orange)? Hard to work with language, vision and other modalities ‘‘is a film based on the novel of the same name by’’(X, Y)

Tim Rockt¨ aschel GPU-accelerated End-to-end Differentiable Planning and Reasoning 12/39

Symbolic Representations

Symbols (constants and predicates) do not share any information: grandpaOf = grandfatherOf No notion of similarity: apple ∼ orange, professorAt ∼ lecturerAt No generalization beyond what can be symbolically inferred: isFruit(apple), apple ∼ organge, isFruit(orange)? Hard to work with language, vision and other modalities ‘‘is a film based on the novel of the same name by’’(X, Y) But... leads to powerful inference mechanisms and proofs for predictions: fatherOf(abe, homer). parentOf(homer, lisa). parentOf(homer, bart). grandfatherOf(X, Y) :– fatherOf(X, Z), parentOf(Z, Y). grandfatherOf(abe, Q)? {Q/lisa}, {Q/bart}

Tim Rockt¨ aschel GPU-accelerated End-to-end Differentiable Planning and Reasoning 12/39

Symbolic Representations

Symbols (constants and predicates) do not share any information: grandpaOf = grandfatherOf No notion of similarity: apple ∼ orange, professorAt ∼ lecturerAt No generalization beyond what can be symbolically inferred: isFruit(apple), apple ∼ organge, isFruit(orange)? Hard to work with language, vision and other modalities ‘‘is a film based on the novel of the same name by’’(X, Y) But... leads to powerful inference mechanisms and proofs for predictions: fatherOf(abe, homer). parentOf(homer, lisa). parentOf(homer, bart). grandfatherOf(X, Y) :– fatherOf(X, Z), parentOf(Z, Y). grandfatherOf(abe, Q)? {Q/lisa}, {Q/bart} Fairly easy to debug and trivial to incorporate domain knowledge: Show to domain expert and let her change/add rules and facts

Tim Rockt¨ aschel GPU-accelerated End-to-end Differentiable Planning and Reasoning 12/39

Neural Representations

Lower-dimensional fixed-length vector representations of symbols (predicates and constants): vapple, vorange, vfatherOf, . . . ∈ Rk

Tim Rockt¨ aschel GPU-accelerated End-to-end Differentiable Planning and Reasoning 13/39

Neural Representations

Lower-dimensional fixed-length vector representations of symbols (predicates and constants): vapple, vorange, vfatherOf, . . . ∈ Rk Can capture similarity and even semantic hierarchy of symbols: vgrandpaOf = vgrandfatherOf, vapple ∼ vorange, vapple < vfruit

Tim Rockt¨ aschel GPU-accelerated End-to-end Differentiable Planning and Reasoning 13/39

Neural Representations

Lower-dimensional fixed-length vector representations of symbols (predicates and constants): vapple, vorange, vfatherOf, . . . ∈ Rk Can capture similarity and even semantic hierarchy of symbols: vgrandpaOf = vgrandfatherOf, vapple ∼ vorange, vapple < vfruit Can be trained from raw task data (e.g. facts in a knowledge base)

Tim Rockt¨ aschel GPU-accelerated End-to-end Differentiable Planning and Reasoning 13/39

Neural Representations

Lower-dimensional fixed-length vector representations of symbols (predicates and constants): vapple, vorange, vfatherOf, . . . ∈ Rk Can capture similarity and even semantic hierarchy of symbols: vgrandpaOf = vgrandfatherOf, vapple ∼ vorange, vapple < vfruit Can be trained from raw task data (e.g. facts in a knowledge base) Can be compositional v‘‘is the father of’’ = RNNθ(vis, vthe, vfather, vof)

Tim Rockt¨ aschel GPU-accelerated End-to-end Differentiable Planning and Reasoning 13/39

Neural Representations

Lower-dimensional fixed-length vector representations of symbols (predicates and constants): vapple, vorange, vfatherOf, . . . ∈ Rk Can capture similarity and even semantic hierarchy of symbols: vgrandpaOf = vgrandfatherOf, vapple ∼ vorange, vapple < vfruit Can be trained from raw task data (e.g. facts in a knowledge base) Can be compositional v‘‘is the father of’’ = RNNθ(vis, vthe, vfather, vof) But... need large amount of training data

Tim Rockt¨ aschel GPU-accelerated End-to-end Differentiable Planning and Reasoning 13/39

Neural Representations

Lower-dimensional fixed-length vector representations of symbols (predicates and constants): vapple, vorange, vfatherOf, . . . ∈ Rk Can capture similarity and even semantic hierarchy of symbols: vgrandpaOf = vgrandfatherOf, vapple ∼ vorange, vapple < vfruit Can be trained from raw task data (e.g. facts in a knowledge base) Can be compositional v‘‘is the father of’’ = RNNθ(vis, vthe, vfather, vof) But... need large amount of training data No direct way of incorporating prior knowledge vgrandfatherOf(X, Y) :– vfatherOf(X, Z), vparentOf(Z, Y).

Tim Rockt¨ aschel GPU-accelerated End-to-end Differentiable Planning and Reasoning 13/39

Machine Learning & Logic

Fuzzy Logic (Zadeh, 1965)

Tim Rockt¨ aschel GPU-accelerated End-to-end Differentiable Planning and Reasoning 14/39

Machine Learning & Logic

Fuzzy Logic (Zadeh, 1965) Probabilistic Logic Programming, e.g.,

Tim Rockt¨ aschel GPU-accelerated End-to-end Differentiable Planning and Reasoning 14/39

Machine Learning & Logic

Fuzzy Logic (Zadeh, 1965) Probabilistic Logic Programming, e.g.,

IBAL (Pfeffer, 2001), BLOG (Milch et al., 2005), Markov Logic Networks (Richardson and Domingos, 2006), ProbLog (De Raedt et al., 2007) . . .

Tim Rockt¨ aschel GPU-accelerated End-to-end Differentiable Planning and Reasoning 14/39

Machine Learning & Logic

Fuzzy Logic (Zadeh, 1965) Probabilistic Logic Programming, e.g.,

IBAL (Pfeffer, 2001), BLOG (Milch et al., 2005), Markov Logic Networks (Richardson and Domingos, 2006), ProbLog (De Raedt et al., 2007) . . .

Inductive Logic Programming, e.g.,

Tim Rockt¨ aschel GPU-accelerated End-to-end Differentiable Planning and Reasoning 14/39

Machine Learning & Logic

Fuzzy Logic (Zadeh, 1965) Probabilistic Logic Programming, e.g.,

IBAL (Pfeffer, 2001), BLOG (Milch et al., 2005), Markov Logic Networks (Richardson and Domingos, 2006), ProbLog (De Raedt et al., 2007) . . .

Inductive Logic Programming, e.g.,

Plotkin (1970), Shapiro (1991), Muggleton (1991), De Raedt (1999) . . .

Tim Rockt¨ aschel GPU-accelerated End-to-end Differentiable Planning and Reasoning 14/39

Machine Learning & Logic

Fuzzy Logic (Zadeh, 1965) Probabilistic Logic Programming, e.g.,

IBAL (Pfeffer, 2001), BLOG (Milch et al., 2005), Markov Logic Networks (Richardson and Domingos, 2006), ProbLog (De Raedt et al., 2007) . . .

Inductive Logic Programming, e.g.,

Plotkin (1970), Shapiro (1991), Muggleton (1991), De Raedt (1999) . . . Statistical Predicate Invention (Kok and Domingos, 2007)

Tim Rockt¨ aschel GPU-accelerated End-to-end Differentiable Planning and Reasoning 14/39

Machine Learning & Logic

Fuzzy Logic (Zadeh, 1965) Probabilistic Logic Programming, e.g.,

IBAL (Pfeffer, 2001), BLOG (Milch et al., 2005), Markov Logic Networks (Richardson and Domingos, 2006), ProbLog (De Raedt et al., 2007) . . .

Inductive Logic Programming, e.g.,

Plotkin (1970), Shapiro (1991), Muggleton (1991), De Raedt (1999) . . . Statistical Predicate Invention (Kok and Domingos, 2007)

Neural-symbolic Connectionism

Tim Rockt¨ aschel GPU-accelerated End-to-end Differentiable Planning and Reasoning 14/39

Machine Learning & Logic

Fuzzy Logic (Zadeh, 1965) Probabilistic Logic Programming, e.g.,

IBAL (Pfeffer, 2001), BLOG (Milch et al., 2005), Markov Logic Networks (Richardson and Domingos, 2006), ProbLog (De Raedt et al., 2007) . . .

Inductive Logic Programming, e.g.,

Plotkin (1970), Shapiro (1991), Muggleton (1991), De Raedt (1999) . . . Statistical Predicate Invention (Kok and Domingos, 2007)

Neural-symbolic Connectionism

Propositional rules: EBL-ANN (Shavlik and Towell, 1989), KBANN (Towell and Shavlik, 1994), C-LIP (Garcez and Zaverucha, 1999)

Tim Rockt¨ aschel GPU-accelerated End-to-end Differentiable Planning and Reasoning 14/39

Machine Learning & Logic

Fuzzy Logic (Zadeh, 1965) Probabilistic Logic Programming, e.g.,

IBAL (Pfeffer, 2001), BLOG (Milch et al., 2005), Markov Logic Networks (Richardson and Domingos, 2006), ProbLog (De Raedt et al., 2007) . . .

Inductive Logic Programming, e.g.,

Plotkin (1970), Shapiro (1991), Muggleton (1991), De Raedt (1999) . . . Statistical Predicate Invention (Kok and Domingos, 2007)

Neural-symbolic Connectionism

Propositional rules: EBL-ANN (Shavlik and Towell, 1989), KBANN (Towell and Shavlik, 1994), C-LIP (Garcez and Zaverucha, 1999) First-order inference (no training of symbol representations): Unification Neural Networks (Holld¨

• bler, 1990; Komendantskaya 2011), SHRUTI

(Shastri, 1992), Neural Prolog (Ding, 1995), CLIP++ (Franca et al. 2014), Lifted Relational Networks (Sourek et al. 2015)

Tim Rockt¨ aschel GPU-accelerated End-to-end Differentiable Planning and Reasoning 14/39

State-of-the-art Neural Link Prediction

livesIn(melinda, seattle)? = f (vlivesIn, vmelinda, vseattle)

Tim Rockt¨ aschel GPU-accelerated End-to-end Differentiable Planning and Reasoning 15/39

State-of-the-art Neural Link Prediction

livesIn(melinda, seattle)? = f (vlivesIn, vmelinda, vseattle) DistMult (Yang et al., 2015) vs, vi, vj ∈ Rk

Tim Rockt¨ aschel GPU-accelerated End-to-end Differentiable Planning and Reasoning 15/39

State-of-the-art Neural Link Prediction

livesIn(melinda, seattle)? = f (vlivesIn, vmelinda, vseattle) DistMult (Yang et al., 2015) vs, vi, vj ∈ Rk

f (vs, vi, vj) = v⊤

s (vi ⊙ vj)

=

• k

vskvikvjk

Tim Rockt¨ aschel GPU-accelerated End-to-end Differentiable Planning and Reasoning 15/39

State-of-the-art Neural Link Prediction

livesIn(melinda, seattle)? = f (vlivesIn, vmelinda, vseattle) DistMult (Yang et al., 2015) vs, vi, vj ∈ Rk

f (vs, vi, vj) = v⊤

s (vi ⊙ vj)

=

• k

vskvikvjk

ComplEx (Trouillon et al., 2016) vs, vi, vj ∈ Ck

Tim Rockt¨ aschel GPU-accelerated End-to-end Differentiable Planning and Reasoning 15/39

State-of-the-art Neural Link Prediction

livesIn(melinda, seattle)? = f (vlivesIn, vmelinda, vseattle) DistMult (Yang et al., 2015) vs, vi, vj ∈ Rk

f (vs, vi, vj) = v⊤

s (vi ⊙ vj)

=

• k

vskvikvjk

ComplEx (Trouillon et al., 2016) vs, vi, vj ∈ Ck

f (vs, vi, vj) = real(vs)⊤(real(vi) ⊙ real(vj)) + real(vs)⊤(imag(vi) ⊙ imag(vj)) + imag(vs)⊤(real(vi) ⊙ imag(vj)) − imag(vs)⊤(imag(vi) ⊙ real(vj))

Tim Rockt¨ aschel GPU-accelerated End-to-end Differentiable Planning and Reasoning 15/39

State-of-the-art Neural Link Prediction

livesIn(melinda, seattle)? = f (vlivesIn, vmelinda, vseattle) DistMult (Yang et al., 2015) vs, vi, vj ∈ Rk

f (vs, vi, vj) = v⊤

s (vi ⊙ vj)

=

• k

vskvikvjk

ComplEx (Trouillon et al., 2016) vs, vi, vj ∈ Ck

f (vs, vi, vj) = real(vs)⊤(real(vi) ⊙ real(vj)) + real(vs)⊤(imag(vi) ⊙ imag(vj)) + imag(vs)⊤(real(vi) ⊙ imag(vj)) − imag(vs)⊤(imag(vi) ⊙ real(vj))

Training Loss

L =

• rs(ei,ej),y ∈ T

−y log (σ(f (vs, vi, vj))) − (1 − y) log (1 − σ(f (vs, vi, vj)))

Tim Rockt¨ aschel GPU-accelerated End-to-end Differentiable Planning and Reasoning 15/39

Differentiable Proving in a Nutshell

Example Knowledge Base:

• 1. fatherOf(abe, homer).
• 2. parentOf(homer, bart).
• 3. grandfatherOf(X, Y) :–

fatherOf(X, Z), parentOf(Z, Y). grandfatherOf abe bart

Tim Rockt¨ aschel GPU-accelerated End-to-end Differentiable Planning and Reasoning 16/39

Differentiable Proving in a Nutshell

Example Knowledge Base:

• 1. fatherOf(abe, homer).
• 2. parentOf(homer, bart).
• 3. grandfatherOf(X, Y) :–

fatherOf(X, Z), parentOf(Z, Y). grandfatherOf abe bart

Tim Rockt¨ aschel GPU-accelerated End-to-end Differentiable Planning and Reasoning 16/39

Differentiable Proving in a Nutshell

Example Knowledge Base:

• 1. fatherOf(abe, homer).
• 2. parentOf(homer, bart).
• 3. grandfatherOf(X, Y) :–

fatherOf(X, Z), parentOf(Z, Y). grandfatherOf abe bart

∅

• 1. fatherOf(abe, homer)

Tim Rockt¨ aschel GPU-accelerated End-to-end Differentiable Planning and Reasoning 16/39

Differentiable Proving in a Nutshell

Example Knowledge Base:

• 1. fatherOf(abe, homer).
• 2. parentOf(homer, bart).
• 3. grandfatherOf(X, Y) :–

fatherOf(X, Z), parentOf(Z, Y). grandfatherOf abe bart

∅

· · ·

X/abe Y/bart

• 1. fatherOf(abe, homer)
• 2. parentOf(homer, bart)
• 3. grandfatherOf(X, Y) :– fatherOf(X, Z), parentOf(Z, Y)

Tim Rockt¨ aschel GPU-accelerated End-to-end Differentiable Planning and Reasoning 16/39

Differentiable Proving in a Nutshell

Example Knowledge Base:

• 1. fatherOf(abe, homer).
• 2. parentOf(homer, bart).
• 3. grandfatherOf(X, Y) :–

fatherOf(X, Z), parentOf(Z, Y). grandfatherOf abe bart

∅

· · ·

X/abe Y/bart

3.1 fatherOf(X, Z) 3.2 parentOf(Z, Y)

• 1. fatherOf(abe, homer)
• 2. parentOf(homer, bart)
• 3. grandfatherOf(X, Y) :– fatherOf(X, Z), parentOf(Z, Y)

Tim Rockt¨ aschel GPU-accelerated End-to-end Differentiable Planning and Reasoning 16/39

Differentiable Proving in a Nutshell

Example Knowledge Base:

• 1. fatherOf(abe, homer).
• 2. parentOf(homer, bart).
• 3. grandfatherOf(X, Y) :–

fatherOf(X, Z), parentOf(Z, Y). grandfatherOf abe bart

∅

· · ·

X/abe Y/bart

3.1 fatherOf(X, Z) 3.2 parentOf(Z, Y) fatherOf abe Z

• 1. fatherOf(abe, homer)
• 2. parentOf(homer, bart)
• 3. grandfatherOf(X, Y) :– fatherOf(X, Z), parentOf(Z, Y)

Tim Rockt¨ aschel GPU-accelerated End-to-end Differentiable Planning and Reasoning 16/39

Differentiable Proving in a Nutshell

Example Knowledge Base:

• 1. fatherOf(abe, homer).
• 2. parentOf(homer, bart).
• 3. grandfatherOf(X, Y) :–

fatherOf(X, Z), parentOf(Z, Y). grandfatherOf abe bart

∅

· · ·

X/abe Y/bart

3.1 fatherOf(X, Z) 3.2 parentOf(Z, Y) fatherOf abe Z

• 1. fatherOf(abe, homer)
• 2. parentOf(homer, bart)
• 3. grandfatherOf(X, Y) :– fatherOf(X, Z), parentOf(Z, Y)

· · · · · ·

• 2. parentOf(homer, bart)
• 3. grandfatherOf(X, Y) :– fatherOf(X, Z), parentOf(Z, Y)

Tim Rockt¨ aschel GPU-accelerated End-to-end Differentiable Planning and Reasoning 16/39

Differentiable Proving in a Nutshell

Example Knowledge Base:

• 1. fatherOf(abe, homer).
• 2. parentOf(homer, bart).
• 3. grandfatherOf(X, Y) :–

fatherOf(X, Z), parentOf(Z, Y). grandfatherOf abe bart

∅

· · ·

X/abe Y/bart

3.1 fatherOf(X, Z) 3.2 parentOf(Z, Y) fatherOf abe Z

• 1. fatherOf(abe, homer)
• 2. parentOf(homer, bart)
• 3. grandfatherOf(X, Y) :– fatherOf(X, Z), parentOf(Z, Y)

X/abe Y/bart Z/homer

· · · · · ·

• 1. fatherOf(abe, homer)
• 2. parentOf(homer, bart)
• 3. grandfatherOf(X, Y) :– fatherOf(X, Z), parentOf(Z, Y)

Tim Rockt¨ aschel GPU-accelerated End-to-end Differentiable Planning and Reasoning 16/39

Differentiable Proving in a Nutshell

Example Knowledge Base:

• 1. fatherOf(abe, homer).
• 2. parentOf(homer, bart).
• 3. grandfatherOf(X, Y) :–

fatherOf(X, Z), parentOf(Z, Y). grandfatherOf abe bart

∅

· · ·

X/abe Y/bart

3.1 fatherOf(X, Z) 3.2 parentOf(Z, Y) fatherOf abe Z

• 1. fatherOf(abe, homer)
• 2. parentOf(homer, bart)
• 3. grandfatherOf(X, Y) :– fatherOf(X, Z), parentOf(Z, Y)

X/abe Y/bart Z/homer

· · · · · ·

• 1. fatherOf(abe, homer)
• 2. parentOf(homer, bart)
• 3. grandfatherOf(X, Y) :– fatherOf(X, Z), parentOf(Z, Y)

3.2 parentOf(Z, Y)

Tim Rockt¨ aschel GPU-accelerated End-to-end Differentiable Planning and Reasoning 16/39

Differentiable Proving in a Nutshell

Example Knowledge Base:

• 1. fatherOf(abe, homer).
• 2. parentOf(homer, bart).
• 3. grandfatherOf(X, Y) :–

fatherOf(X, Z), parentOf(Z, Y). grandfatherOf abe bart

∅

· · ·

X/abe Y/bart

3.1 fatherOf(X, Z) 3.2 parentOf(Z, Y) fatherOf abe Z

• 1. fatherOf(abe, homer)
• 2. parentOf(homer, bart)
• 3. grandfatherOf(X, Y) :– fatherOf(X, Z), parentOf(Z, Y)

X/abe Y/bart Z/homer

· · · · · ·

• 1. fatherOf(abe, homer)
• 2. parentOf(homer, bart)
• 3. grandfatherOf(X, Y) :– fatherOf(X, Z), parentOf(Z, Y)

3.2 parentOf(Z, Y) parentOf homer bart

Tim Rockt¨ aschel GPU-accelerated End-to-end Differentiable Planning and Reasoning 16/39

Differentiable Proving in a Nutshell

Example Knowledge Base:

• 1. fatherOf(abe, homer).
• 2. parentOf(homer, bart).
• 3. grandfatherOf(X, Y) :–

fatherOf(X, Z), parentOf(Z, Y). grandfatherOf abe bart

∅

· · ·

X/abe Y/bart

3.1 fatherOf(X, Z) 3.2 parentOf(Z, Y) fatherOf abe Z

• 1. fatherOf(abe, homer)
• 2. parentOf(homer, bart)
• 3. grandfatherOf(X, Y) :– fatherOf(X, Z), parentOf(Z, Y)

X/abe Y/bart Z/homer

· · · · · ·

• 1. fatherOf(abe, homer)
• 2. parentOf(homer, bart)
• 3. grandfatherOf(X, Y) :– fatherOf(X, Z), parentOf(Z, Y)

3.2 parentOf(Z, Y) parentOf homer bart

X/abe Y/bart Z/homer X/abe Y/bart Z/homer

· · ·

• 1. fatherOf(abe, homer)
• 3. grandfatherOf(X, Y) :– fatherOf(X, Z), parentOf(Z, Y)
• 2. parentOf(homer, bart)

Tim Rockt¨ aschel GPU-accelerated End-to-end Differentiable Planning and Reasoning 16/39

Proof States S = (Ψ, ρ)

Substitution set Ψ constructed in the proof so far

Tim Rockt¨ aschel GPU-accelerated End-to-end Differentiable Planning and Reasoning 17/39

Proof States S = (Ψ, ρ)

Substitution set Ψ constructed in the proof so far Neural network ρ that outputs a real-valued proof success score

Tim Rockt¨ aschel GPU-accelerated End-to-end Differentiable Planning and Reasoning 17/39

Proof States S = (Ψ, ρ)

Substitution set Ψ constructed in the proof so far Neural network ρ that outputs a real-valued proof success score

Tim Rockt¨ aschel GPU-accelerated End-to-end Differentiable Planning and Reasoning 17/39

Proof States S = (Ψ, ρ)

Substitution set Ψ constructed in the proof so far Neural network ρ that outputs a real-valued proof success score X/Q Y/bart SΨ Sρ

Tim Rockt¨ aschel GPU-accelerated End-to-end Differentiable Planning and Reasoning 17/39

Proof Modules

unifyθ, orK

θ, andK θ

Modular construction of differentiable prover

Tim Rockt¨ aschel GPU-accelerated End-to-end Differentiable Planning and Reasoning 18/39

Proof Modules

unifyθ, orK

θ, andK θ

Modular construction of differentiable prover Discrete objects (rules, atoms etc.) are used to instantiate proof modules

Tim Rockt¨ aschel GPU-accelerated End-to-end Differentiable Planning and Reasoning 18/39

Proof Modules

unifyθ, orK

θ, andK θ

Modular construction of differentiable prover Discrete objects (rules, atoms etc.) are used to instantiate proof modules Modules transform proof states into new proof states

Tim Rockt¨ aschel GPU-accelerated End-to-end Differentiable Planning and Reasoning 18/39

Proof Modules

unifyθ, orK

θ, andK θ

Modular construction of differentiable prover Discrete objects (rules, atoms etc.) are used to instantiate proof modules Modules transform proof states into new proof states X/Q Y/bart SΨ Sρ

Tim Rockt¨ aschel GPU-accelerated End-to-end Differentiable Planning and Reasoning 18/39

Proof Modules

unifyθ, orK

θ, andK θ

Modular construction of differentiable prover Discrete objects (rules, atoms etc.) are used to instantiate proof modules Modules transform proof states into new proof states X/Q Y/bart SΨ Sρ X/Q Y/bart Z/homer S′

Ψ

S′

ρ Tim Rockt¨ aschel GPU-accelerated End-to-end Differentiable Planning and Reasoning 18/39

Proof Modules

unifyθ, orK

θ, andK θ

Modular construction of differentiable prover Discrete objects (rules, atoms etc.) are used to instantiate proof modules Modules transform proof states into new proof states X/Q Y/bart SΨ Sρ X/Q Y/bart Z/homer S′

Ψ

S′

ρ Tim Rockt¨ aschel GPU-accelerated End-to-end Differentiable Planning and Reasoning 18/39

Unification Module

unify takes two atoms represented as lists of terms and an upstream proof state, and maps these to a new proof state (substitution set and proof success)

Tim Rockt¨ aschel GPU-accelerated End-to-end Differentiable Planning and Reasoning 19/39

Unification Module

unify takes two atoms represented as lists of terms and an upstream proof state, and maps these to a new proof state (substitution set and proof success)

• 1. unifyθ([ ], [ ], S) = S

Tim Rockt¨ aschel GPU-accelerated End-to-end Differentiable Planning and Reasoning 19/39

Unification Module

unify takes two atoms represented as lists of terms and an upstream proof state, and maps these to a new proof state (substitution set and proof success)

• 1. unifyθ([ ], [ ], S) = S
• 2. unifyθ([ ], G, S) = FAIL

Tim Rockt¨ aschel GPU-accelerated End-to-end Differentiable Planning and Reasoning 19/39

Unification Module

unify takes two atoms represented as lists of terms and an upstream proof state, and maps these to a new proof state (substitution set and proof success)

• 1. unifyθ([ ], [ ], S) = S
• 2. unifyθ([ ], G, S) = FAIL
• 3. unifyθ(H, [ ], S) = FAIL

Tim Rockt¨ aschel GPU-accelerated End-to-end Differentiable Planning and Reasoning 19/39

Unification Module

unify takes two atoms represented as lists of terms and an upstream proof state, and maps these to a new proof state (substitution set and proof success)

• 1. unifyθ([ ], [ ], S) = S
• 2. unifyθ([ ], G, S) = FAIL
• 3. unifyθ(H, [ ], S) = FAIL
• 4. unifyθ(h :: H, g :: G, S) = unifyθ(H, G, S′)

Tim Rockt¨ aschel GPU-accelerated End-to-end Differentiable Planning and Reasoning 19/39

Unification Module

unify takes two atoms represented as lists of terms and an upstream proof state, and maps these to a new proof state (substitution set and proof success)

• 1. unifyθ([ ], [ ], S) = S
• 2. unifyθ([ ], G, S) = FAIL
• 3. unifyθ(H, [ ], S) = FAIL
• 4. unifyθ(h :: H, g :: G, S) = unifyθ(H, G, S′)

S′

Ψ = SΨ ∪

{h/g}

if h ∈ V {g/h} if g ∈ V, h ∈ V ∅

• therwise
• Tim Rockt¨

aschel GPU-accelerated End-to-end Differentiable Planning and Reasoning 19/39

Unification Module

unify takes two atoms represented as lists of terms and an upstream proof state, and maps these to a new proof state (substitution set and proof success)

• 1. unifyθ([ ], [ ], S) = S
• 2. unifyθ([ ], G, S) = FAIL
• 3. unifyθ(H, [ ], S) = FAIL
• 4. unifyθ(h :: H, g :: G, S) = unifyθ(H, G, S′)

S′

Ψ = SΨ ∪

{h/g}

if h ∈ V {g/h} if g ∈ V, h ∈ V ∅

• therwise
• S′

ρ = min

• Sρ,
• exp (−θh: − θg:2)

if h ∈ V, g ∈ V 1

• therwise
• Tim Rockt¨

aschel GPU-accelerated End-to-end Differentiable Planning and Reasoning 19/39

Unification Module

unify takes two atoms represented as lists of terms and an upstream proof state, and maps these to a new proof state (substitution set and proof success)

• 1. unifyθ([ ], [ ], S) = S
• 2. unifyθ([ ], G, S) = FAIL
• 3. unifyθ(H, [ ], S) = FAIL
• 4. unifyθ(h :: H, g :: G, S) = unifyθ(H, G, S′)

S′

Ψ = SΨ ∪

{h/g}

if h ∈ V {g/h} if g ∈ V, h ∈ V ∅

• therwise
• S′

ρ = min

• Sρ,
• exp (−θh: − θg:2)

if h ∈ V, g ∈ V 1

• therwise
• Example:

unifyθ([grandpaOf, abe, bart], [s, Q, i], (∅, 0.7)) =

• {Q/abe}, min(0.7, exp(−θgrandpaOf: − θs:2), exp(−θbart: − θi:2))
• Tim Rockt¨

aschel GPU-accelerated End-to-end Differentiable Planning and Reasoning 19/39

OR Module

• 1. orK

θ(G, d, S) = [S′ | S′ ∈ andK θ(B, d, unifyθ(H, G, S)), H :– B ∈ K]

G is a goal atom, d is the maximum proof depth, and H :– B is a rule

Tim Rockt¨ aschel GPU-accelerated End-to-end Differentiable Planning and Reasoning 20/39

OR Module

• 1. orK

θ(G, d, S) = [S′ | S′ ∈ andK θ(B, d, unifyθ(H, G, S)), H :– B ∈ K]

G is a goal atom, d is the maximum proof depth, and H :– B is a rule

• r iterates through all rules (including rules with an empty body, i.e., facts) and

unifies the goal with the respective rule head

Tim Rockt¨ aschel GPU-accelerated End-to-end Differentiable Planning and Reasoning 20/39

OR Module

• 1. orK

θ(G, d, S) = [S′ | S′ ∈ andK θ(B, d, unifyθ(H, G, S)), H :– B ∈ K]

G is a goal atom, d is the maximum proof depth, and H :– B is a rule

• r iterates through all rules (including rules with an empty body, i.e., facts) and

unifies the goal with the respective rule head If unification succeeds, it instantiates an and module to prove all atoms in the body of the rule.

Tim Rockt¨ aschel GPU-accelerated End-to-end Differentiable Planning and Reasoning 20/39

OR Module

• 1. orK

θ(G, d, S) = [S′ | S′ ∈ andK θ(B, d, unifyθ(H, G, S)), H :– B ∈ K]

G is a goal atom, d is the maximum proof depth, and H :– B is a rule

• r iterates through all rules (including rules with an empty body, i.e., facts) and

unifies the goal with the respective rule head If unification succeeds, it instantiates an and module to prove all atoms in the body of the rule. In other words, it is translating goals into subgoals using rules, e.g., grandfatherOf(Q, bart) is translated into subgoals fatherOf(Q, Z) and parentOf(Z, bart) using the rule grandfatherOf(X, Y) :– fatherOf(X, Z), parentOf(Z, Y)

Tim Rockt¨ aschel GPU-accelerated End-to-end Differentiable Planning and Reasoning 20/39

OR Module

• 1. orK

θ(G, d, S) = [S′ | S′ ∈ andK θ(B, d, unifyθ(H, G, S)), H :– B ∈ K]

G is a goal atom, d is the maximum proof depth, and H :– B is a rule

• r iterates through all rules (including rules with an empty body, i.e., facts) and

unifies the goal with the respective rule head If unification succeeds, it instantiates an and module to prove all atoms in the body of the rule. In other words, it is translating goals into subgoals using rules, e.g., grandfatherOf(Q, bart) is translated into subgoals fatherOf(Q, Z) and parentOf(Z, bart) using the rule grandfatherOf(X, Y) :– fatherOf(X, Z), parentOf(Z, Y) Example:

• rK

θ ([grandfatherOf, Q, bart], d, S) =

[S′|S′ ∈ andK

θ ([[fatherOf, X, Z], [parentOf, Z, Y]], d, ({X/Q, Y/bart}, ˆ

Sρ)

• result of unifyθ

), . . .]

Tim Rockt¨ aschel GPU-accelerated End-to-end Differentiable Planning and Reasoning 20/39

AND Module

• 1. andK

θ (G, d, FAIL) = FAIL

Tim Rockt¨ aschel GPU-accelerated End-to-end Differentiable Planning and Reasoning 21/39

AND Module

• 1. andK

θ (G, d, FAIL) = FAIL

• 2. andK

θ (G, 0, S) = FAIL

Tim Rockt¨ aschel GPU-accelerated End-to-end Differentiable Planning and Reasoning 21/39

AND Module

• 1. andK

θ (G, d, FAIL) = FAIL

• 2. andK

θ (G, 0, S) = FAIL

• 3. andK

θ ([ ], d, S) = S

Tim Rockt¨ aschel GPU-accelerated End-to-end Differentiable Planning and Reasoning 21/39

AND Module

• 1. andK

θ (G, d, FAIL) = FAIL

• 2. andK

θ (G, 0, S) = FAIL

• 3. andK

θ ([ ], d, S) = S

• 4. andK

θ (G :: G, d, S) = [S′′ | S′′ ∈ andK θ (G, d, S′), S′ ∈ orK θ (substitute(G, S), d − 1, S)]

Tim Rockt¨ aschel GPU-accelerated End-to-end Differentiable Planning and Reasoning 21/39

AND Module

• 1. andK

θ (G, d, FAIL) = FAIL

• 2. andK

θ (G, 0, S) = FAIL

• 3. andK

θ ([ ], d, S) = S

• 4. andK

θ (G :: G, d, S) = [S′′ | S′′ ∈ andK θ (G, d, S′), S′ ∈ orK θ (substitute(G, S), d − 1, S)]

Example:

andK

θ ([[fatherOf, X, Z], [parentOf, Z, Y]], d, ({X/Q, Y/bart}, ˆ

Sρ)

• result of unifyθ in orK

θ

) = [S′′|S′′ ∈ andK

θ ([[parentOf, Z, Y]], d, S′), S′ ∈ orK θ ([fatherOf, Q, Z]

• result of substitute

, d − 1, S)]

Tim Rockt¨ aschel GPU-accelerated End-to-end Differentiable Planning and Reasoning 21/39

Proof Aggregation

Goal G = [s, i, j] where s is the index of a predicate symbol and i, j are indices of constant symbols d maximum proof depth and proof start state (∅, 1) ntpK

θ(G, d) =

max

S ∈ orK

θ (G,d,(∅,1))

S=FAIL

Sρ

Tim Rockt¨ aschel GPU-accelerated End-to-end Differentiable Planning and Reasoning 22/39

Proof Aggregation

Goal G = [s, i, j] where s is the index of a predicate symbol and i, j are indices of constant symbols d maximum proof depth and proof start state (∅, 1) ntpK

θ(G, d) =

max

S ∈ orK

θ (G,d,(∅,1))

S=FAIL

Sρ

Training Loss

LntpK

θ =

• (G,y) ∈ T

−y log(ntpK

θ(G, d)) − (1 − y) log(1 − ntpK θ(G, d)) Tim Rockt¨ aschel GPU-accelerated End-to-end Differentiable Planning and Reasoning 22/39

• rK

θ([s, i, j], 2, (∅, 1))

Example Knowledge Base:

• 1. fatherOf(abe, homer).
• 2. parentOf(homer, bart).
• 3. grandfatherOf(X, Y) :–

fatherOf(X, Z), parentOf(Z, Y).

Tim Rockt¨ aschel GPU-accelerated End-to-end Differentiable Planning and Reasoning 23/39

• rK

θ([s, i, j], 2, (∅, 1))

unifyθ([fatherOf, abe, homer], [s, i, j], (∅, 1)) unifyθ([grandfatherOf, X, Y], [s, i, j], (∅, 1)) 1. 3. S1 = (∅, ρ1) S2 = (∅, ρ2) 2. . . . Example Knowledge Base:

• 1. fatherOf(abe, homer).
• 2. parentOf(homer, bart).
• 3. grandfatherOf(X, Y) :–

fatherOf(X, Z), parentOf(Z, Y).

Tim Rockt¨ aschel GPU-accelerated End-to-end Differentiable Planning and Reasoning 23/39

• rK

θ([s, i, j], 2, (∅, 1))

unifyθ([fatherOf, abe, homer], [s, i, j], (∅, 1)) unifyθ([grandfatherOf, X, Y], [s, i, j], (∅, 1)) 1. 3. S1 = (∅, ρ1) S2 = (∅, ρ2) 2. . . . andK

θ([[fatherOf, X, Z], [parentOf, Z, Y]], 2, S3)

S3 = ({X/i, Y/j}, ρ3) Example Knowledge Base:

• 1. fatherOf(abe, homer).
• 2. parentOf(homer, bart).
• 3. grandfatherOf(X, Y) :–

fatherOf(X, Z), parentOf(Z, Y).

Tim Rockt¨ aschel GPU-accelerated End-to-end Differentiable Planning and Reasoning 23/39

• rK

θ([s, i, j], 2, (∅, 1))

unifyθ([fatherOf, abe, homer], [s, i, j], (∅, 1)) unifyθ([grandfatherOf, X, Y], [s, i, j], (∅, 1)) 1. 3. S1 = (∅, ρ1) S2 = (∅, ρ2) 2. . . . andK

θ([[fatherOf, X, Z], [parentOf, Z, Y]], 2, S3)

S3 = ({X/i, Y/j}, ρ3)

• rK

θ([fatherOf, i, Z], 1, S3)

substitute Example Knowledge Base:

• 1. fatherOf(abe, homer).
• 2. parentOf(homer, bart).
• 3. grandfatherOf(X, Y) :–

fatherOf(X, Z), parentOf(Z, Y).

Tim Rockt¨ aschel GPU-accelerated End-to-end Differentiable Planning and Reasoning 23/39

• rK

θ([s, i, j], 2, (∅, 1))

unifyθ([fatherOf, abe, homer], [s, i, j], (∅, 1)) unifyθ([grandfatherOf, X, Y], [s, i, j], (∅, 1)) 1. 3. S1 = (∅, ρ1) S2 = (∅, ρ2) 2. . . . andK

θ([[fatherOf, X, Z], [parentOf, Z, Y]], 2, S3)

S3 = ({X/i, Y/j}, ρ3)

• rK

θ([fatherOf, i, Z], 1, S3)

substitute unifyθ([fatherOf, abe, homer], [fatherOf, i, Z], S3) unifyθ([parentOf, homer, bart], [fatherOf, i, Z], S3) 1. 2. S33 = FAIL 3. . . . Example Knowledge Base:

• 1. fatherOf(abe, homer).
• 2. parentOf(homer, bart).
• 3. grandfatherOf(X, Y) :–

fatherOf(X, Z), parentOf(Z, Y).

Tim Rockt¨ aschel GPU-accelerated End-to-end Differentiable Planning and Reasoning 23/39

• rK

θ([s, i, j], 2, (∅, 1))

unifyθ([fatherOf, abe, homer], [s, i, j], (∅, 1)) unifyθ([grandfatherOf, X, Y], [s, i, j], (∅, 1)) 1. 3. S1 = (∅, ρ1) S2 = (∅, ρ2) 2. . . . andK

θ([[fatherOf, X, Z], [parentOf, Z, Y]], 2, S3)

S3 = ({X/i, Y/j}, ρ3)

• rK

θ([fatherOf, i, Z], 1, S3)

substitute unifyθ([fatherOf, abe, homer], [fatherOf, i, Z], S3) unifyθ([parentOf, homer, bart], [fatherOf, i, Z], S3) 1. 2. S33 = FAIL 3. . . . andK

θ([parentOf, Z, Y], 2, S31)

S31 = ({X/i, Y/j, Z/homer}, ρ31) Example Knowledge Base:

• 1. fatherOf(abe, homer).
• 2. parentOf(homer, bart).
• 3. grandfatherOf(X, Y) :–

fatherOf(X, Z), parentOf(Z, Y).

Tim Rockt¨ aschel GPU-accelerated End-to-end Differentiable Planning and Reasoning 23/39

• rK

θ([s, i, j], 2, (∅, 1))

unifyθ([fatherOf, abe, homer], [s, i, j], (∅, 1)) unifyθ([grandfatherOf, X, Y], [s, i, j], (∅, 1)) 1. 3. S1 = (∅, ρ1) S2 = (∅, ρ2) 2. . . . andK

θ([[fatherOf, X, Z], [parentOf, Z, Y]], 2, S3)

S3 = ({X/i, Y/j}, ρ3)

• rK

θ([fatherOf, i, Z], 1, S3)

substitute unifyθ([fatherOf, abe, homer], [fatherOf, i, Z], S3) unifyθ([parentOf, homer, bart], [fatherOf, i, Z], S3) 1. 2. S33 = FAIL 3. . . . andK

θ([parentOf, Z, Y], 2, S31)

S31 = ({X/i, Y/j, Z/homer}, ρ31)

• rK

θ([parentOf, homer, j], 1, S31)

substitute Example Knowledge Base:

• 1. fatherOf(abe, homer).
• 2. parentOf(homer, bart).
• 3. grandfatherOf(X, Y) :–

fatherOf(X, Z), parentOf(Z, Y).

Tim Rockt¨ aschel GPU-accelerated End-to-end Differentiable Planning and Reasoning 23/39

• rK

θ([s, i, j], 2, (∅, 1))

unifyθ([fatherOf, abe, homer], [s, i, j], (∅, 1)) unifyθ([grandfatherOf, X, Y], [s, i, j], (∅, 1)) 1. 3. S1 = (∅, ρ1) S2 = (∅, ρ2) 2. . . . andK

θ([[fatherOf, X, Z], [parentOf, Z, Y]], 2, S3)

S3 = ({X/i, Y/j}, ρ3)

• rK

θ([fatherOf, i, Z], 1, S3)

substitute unifyθ([fatherOf, abe, homer], [fatherOf, i, Z], S3) unifyθ([parentOf, homer, bart], [fatherOf, i, Z], S3) 1. 2. S33 = FAIL 3. . . . andK

θ([parentOf, Z, Y], 2, S31)

S31 = ({X/i, Y/j, Z/homer}, ρ31)

• rK

θ([parentOf, homer, j], 1, S31)

substitute S311 = ({X/i, Y/j, Z/homer}, ρ311) S312 = ({X/i, Y/j, Z/homer}, ρ312) S313 = FAIL 1. . . . 2. . . . 3.. . . Example Knowledge Base:

• 1. fatherOf(abe, homer).
• 2. parentOf(homer, bart).
• 3. grandfatherOf(X, Y) :–

fatherOf(X, Z), parentOf(Z, Y).

Tim Rockt¨ aschel GPU-accelerated End-to-end Differentiable Planning and Reasoning 23/39

• rK

θ([s, i, j], 2, (∅, 1))

unifyθ([fatherOf, abe, homer], [s, i, j], (∅, 1)) unifyθ([grandfatherOf, X, Y], [s, i, j], (∅, 1)) 1. 3. S1 = (∅, ρ1) S2 = (∅, ρ2) 2. . . . andK

θ([[fatherOf, X, Z], [parentOf, Z, Y]], 2, S3)

S3 = ({X/i, Y/j}, ρ3)

• rK

θ([fatherOf, i, Z], 1, S3)

substitute unifyθ([fatherOf, abe, homer], [fatherOf, i, Z], S3) unifyθ([parentOf, homer, bart], [fatherOf, i, Z], S3) 1. 2. S33 = FAIL 3. . . . andK

θ([parentOf, Z, Y], 2, S31)

S31 = ({X/i, Y/j, Z/homer}, ρ31)

• rK

θ([parentOf, homer, j], 1, S31)

substitute S311 = ({X/i, Y/j, Z/homer}, ρ311) S312 = ({X/i, Y/j, Z/homer}, ρ312) S313 = FAIL 1. . . . 2. . . . 3.. . . andK

θ([parentOf, Z, Y], 2, S32)

S32 = ({X/i, Y/j, Z/bart}, ρ32) Example Knowledge Base:

• 1. fatherOf(abe, homer).
• 2. parentOf(homer, bart).
• 3. grandfatherOf(X, Y) :–

fatherOf(X, Z), parentOf(Z, Y).

Tim Rockt¨ aschel GPU-accelerated End-to-end Differentiable Planning and Reasoning 23/39

• rK

θ([s, i, j], 2, (∅, 1))

unifyθ([fatherOf, abe, homer], [s, i, j], (∅, 1)) unifyθ([grandfatherOf, X, Y], [s, i, j], (∅, 1)) 1. 3. S1 = (∅, ρ1) S2 = (∅, ρ2) 2. . . . andK

θ([[fatherOf, X, Z], [parentOf, Z, Y]], 2, S3)

S3 = ({X/i, Y/j}, ρ3)

• rK

θ([fatherOf, i, Z], 1, S3)

substitute unifyθ([fatherOf, abe, homer], [fatherOf, i, Z], S3) unifyθ([parentOf, homer, bart], [fatherOf, i, Z], S3) 1. 2. S33 = FAIL 3. . . . andK

θ([parentOf, Z, Y], 2, S31)

S31 = ({X/i, Y/j, Z/homer}, ρ31)

• rK

θ([parentOf, homer, j], 1, S31)

substitute S311 = ({X/i, Y/j, Z/homer}, ρ311) S312 = ({X/i, Y/j, Z/homer}, ρ312) S313 = FAIL 1. . . . 2. . . . 3.. . . andK

θ([parentOf, Z, Y], 2, S32)

S32 = ({X/i, Y/j, Z/bart}, ρ32)

• rK

θ([parentOf, bart, j], 1, S32)

substitute Example Knowledge Base:

• 1. fatherOf(abe, homer).
• 2. parentOf(homer, bart).
• 3. grandfatherOf(X, Y) :–

fatherOf(X, Z), parentOf(Z, Y).

Tim Rockt¨ aschel GPU-accelerated End-to-end Differentiable Planning and Reasoning 23/39

• rK

θ([s, i, j], 2, (∅, 1))

unifyθ([fatherOf, abe, homer], [s, i, j], (∅, 1)) unifyθ([grandfatherOf, X, Y], [s, i, j], (∅, 1)) 1. 3. S1 = (∅, ρ1) S2 = (∅, ρ2) 2. . . . andK

θ([[fatherOf, X, Z], [parentOf, Z, Y]], 2, S3)

S3 = ({X/i, Y/j}, ρ3)

• rK

θ([fatherOf, i, Z], 1, S3)

substitute unifyθ([fatherOf, abe, homer], [fatherOf, i, Z], S3) unifyθ([parentOf, homer, bart], [fatherOf, i, Z], S3) 1. 2. S33 = FAIL 3. . . . andK

θ([parentOf, Z, Y], 2, S31)

S31 = ({X/i, Y/j, Z/homer}, ρ31)

• rK

θ([parentOf, homer, j], 1, S31)

substitute S311 = ({X/i, Y/j, Z/homer}, ρ311) S312 = ({X/i, Y/j, Z/homer}, ρ312) S313 = FAIL 1. . . . 2. . . . 3.. . . andK

θ([parentOf, Z, Y], 2, S32)

S32 = ({X/i, Y/j, Z/bart}, ρ32)

• rK

θ([parentOf, bart, j], 1, S32)

substitute S321 = ({X/i, Y/j, Z/bart}, ρ321) S322 = ({X/i, Y/j, Z/bart}, ρ322) S323 = FAIL 1.. . . 2. . . . 3. . . . Example Knowledge Base:

• 1. fatherOf(abe, homer).
• 2. parentOf(homer, bart).
• 3. grandfatherOf(X, Y) :–

fatherOf(X, Z), parentOf(Z, Y).

Tim Rockt¨ aschel GPU-accelerated End-to-end Differentiable Planning and Reasoning 23/39

• rK

θ([s, i, j], 2, (∅, 1))

unifyθ([fatherOf, abe, homer], [s, i, j], (∅, 1)) unifyθ([grandfatherOf, X, Y], [s, i, j], (∅, 1)) 1. 3. S1 = (∅, ρ1) S2 = (∅, ρ2) 2. . . . andK

θ([[fatherOf, X, Z], [parentOf, Z, Y]], 2, S3)

S3 = ({X/i, Y/j}, ρ3)

• rK

θ([fatherOf, i, Z], 1, S3)

substitute unifyθ([fatherOf, abe, homer], [fatherOf, i, Z], S3) unifyθ([parentOf, homer, bart], [fatherOf, i, Z], S3) 1. 2. S33 = FAIL 3. . . . andK

θ([parentOf, Z, Y], 2, S31)

S31 = ({X/i, Y/j, Z/homer}, ρ31)

• rK

θ([parentOf, homer, j], 1, S31)

substitute S311 = ({X/i, Y/j, Z/homer}, ρ311) S312 = ({X/i, Y/j, Z/homer}, ρ312) S313 = FAIL 1. . . . 2. . . . 3.. . . andK

θ([parentOf, Z, Y], 2, S32)

S32 = ({X/i, Y/j, Z/bart}, ρ32)

• rK

θ([parentOf, bart, j], 1, S32)

substitute S321 = ({X/i, Y/j, Z/bart}, ρ321) S322 = ({X/i, Y/j, Z/bart}, ρ322) S323 = FAIL 1.. . . 2. . . . 3. . . . Example Knowledge Base:

• 1. fatherOf(abe, homer).
• 2. parentOf(homer, bart).
• 3. grandfatherOf(X, Y) :–

fatherOf(X, Z), parentOf(Z, Y).

Tim Rockt¨ aschel GPU-accelerated End-to-end Differentiable Planning and Reasoning 23/39

Batch Proving

A ∈ RN×k matrix of N subsymbolic representations B ∈ RM×k matrix of M other subsymbolic representations

exp     −

• 

     k

i=1 A2 1i

. . . k

i=1 A2 Ni

   1⊤

M

   +    1N    k

i=1 B2 1i

. . . k

i=1 B2 Mi

  

⊤

   − 2AB⊤      ∈ RN×M

where 1N and 1M are vectors of N and M ones respectively, and the square root is taken element-wise.

Tim Rockt¨ aschel GPU-accelerated End-to-end Differentiable Planning and Reasoning 24/39

Calculation on GPU

Q

parentOf dadOf homer abe

Tim Rockt¨ aschel GPU-accelerated End-to-end Differentiable Planning and Reasoning 25/39

Calculation on GPU

Q

parentOf dadOf homer abe fatherOf parentOf grandmaOf abe homer mona homer bart lisa

Tim Rockt¨ aschel GPU-accelerated End-to-end Differentiable Planning and Reasoning 25/39

Calculation on GPU

Q

parentOf dadOf homer abe fatherOf parentOf grandmaOf abe homer mona homer bart lisa unify unify

Tim Rockt¨ aschel GPU-accelerated End-to-end Differentiable Planning and Reasoning 25/39

Calculation on GPU

Q Q /

parentOf dadOf homer abe fatherOf parentOf grandmaOf abe homer mona homer bart lisa homer bart lisa homer bart lisa unify unify unify (symbolic)

Tim Rockt¨ aschel GPU-accelerated End-to-end Differentiable Planning and Reasoning 25/39

Calculation on GPU

Q Q /

parentOf dadOf homer abe fatherOf parentOf grandmaOf abe homer mona homer bart lisa homer bart lisa homer bart lisa unify unify unify (symbolic)

Tim Rockt¨ aschel GPU-accelerated End-to-end Differentiable Planning and Reasoning 25/39

Neural Inductive Logic Programming

1 vfatherOf(vabe, vhomer). 2 vparentOf(vhomer, vlisa). 3 vparentOf(vhomer, vbart). 4 vgrandpaOf(vabe, vlisa). 5 vgrandfatherOf(vabe, vmaggie).

Tim Rockt¨ aschel GPU-accelerated End-to-end Differentiable Planning and Reasoning 26/39

Neural Inductive Logic Programming

1 vfatherOf(vabe, vhomer). 2 vparentOf(vhomer, vlisa). 3 vparentOf(vhomer, vbart). 4 vgrandpaOf(vabe, vlisa). 5 vgrandfatherOf(vabe, vmaggie). 6 θ1(X1, Y1) :– θ2(X1, Z1), θ3(Z1, Y1). 7 θ4(X2, Y2) :– θ5(X2, Y2).

Tim Rockt¨ aschel GPU-accelerated End-to-end Differentiable Planning and Reasoning 26/39

Neural Inductive Logic Programming

1 vfatherOf(vabe, vhomer). 2 vparentOf(vhomer, vlisa). 3 vparentOf(vhomer, vbart). 4 vgrandpaOf(vabe, vlisa). 5 vgrandfatherOf(vabe, vmaggie). 6 θ1(X1, Y1) :– θ2(X1, Z1), θ3(Z1, Y1). 7 θ4(X2, Y2) :– θ5(X2, Y2).

Decoding Induced Rules Find closest representations of known predicate

Tim Rockt¨ aschel GPU-accelerated End-to-end Differentiable Planning and Reasoning 26/39

Neural Inductive Logic Programming

1 vfatherOf(vabe, vhomer). 2 vparentOf(vhomer, vlisa). 3 vparentOf(vhomer, vbart). 4 vgrandpaOf(vabe, vlisa). 5 vgrandfatherOf(vabe, vmaggie). 6 θ1(X1, Y1) :– θ2(X1, Z1), θ3(Z1, Y1). 7 θ4(X2, Y2) :– θ5(X2, Y2).

Decoding Induced Rules Find closest representations of known predicate Take minimum RBF similarity as rule confidence

Tim Rockt¨ aschel GPU-accelerated End-to-end Differentiable Planning and Reasoning 26/39

Neural Inductive Logic Programming

1 vfatherOf(vabe, vhomer). 2 vparentOf(vhomer, vlisa). 3 vparentOf(vhomer, vbart). 4 vgrandpaOf(vabe, vlisa). 5 vgrandfatherOf(vabe, vmaggie). 6 θ1(X1, Y1) :– θ2(X1, Z1), θ3(Z1, Y1). 7 θ4(X2, Y2) :– θ5(X2, Y2).

Decoding Induced Rules Find closest representations of known predicate Take minimum RBF similarity as rule confidence Rule confidence is an upper bound on the proof success that can be achieved when applying the rule

Tim Rockt¨ aschel GPU-accelerated End-to-end Differentiable Planning and Reasoning 26/39

Experiments

Benchmark Knowledge Bases: Kinship, Nations, UMLS (Kok and Domingos, 2007), and Countries (Bouchard et al., 2015) Test Country Train Country Region Subregion

neighborOf locatedIn locatedIn locatedIn locatedIn locatedIn locatedIn locatedIn locatedIn

Test Country Train Country Region Subregion

neighborOf locatedIn locatedIn locatedIn locatedIn

Tim Rockt¨ aschel GPU-accelerated End-to-end Differentiable Planning and Reasoning 27/39

Experiments

Benchmark Knowledge Bases: Kinship, Nations, UMLS (Kok and Domingos, 2007), and Countries (Bouchard et al., 2015) Test Country Train Country Region Subregion

neighborOf locatedIn locatedIn locatedIn locatedIn locatedIn locatedIn locatedIn

Test Country Train Country Region Subregion

neighborOf locatedIn locatedIn locatedIn locatedIn

Tim Rockt¨ aschel GPU-accelerated End-to-end Differentiable Planning and Reasoning 27/39

Experiments

Benchmark Knowledge Bases: Kinship, Nations, UMLS (Kok and Domingos, 2007), and Countries (Bouchard et al., 2015) Test Country Train Country Region Subregion

neighborOf locatedIn locatedIn locatedIn locatedIn locatedIn locatedIn

Test Country Train Country Region Subregion

neighborOf locatedIn locatedIn locatedIn locatedIn

Tim Rockt¨ aschel GPU-accelerated End-to-end Differentiable Planning and Reasoning 27/39

Experiments

Benchmark Knowledge Bases: Kinship, Nations, UMLS (Kok and Domingos, 2007), and Countries (Bouchard et al., 2015) Test Country Train Country Region Subregion

neighborOf locatedIn locatedIn locatedIn locatedIn locatedIn

Test Country Train Country Region Subregion

neighborOf locatedIn locatedIn locatedIn locatedIn

Tim Rockt¨ aschel GPU-accelerated End-to-end Differentiable Planning and Reasoning 27/39

Details

Models implemented in TensorFlow

Tim Rockt¨ aschel GPU-accelerated End-to-end Differentiable Planning and Reasoning 28/39

Details

Models implemented in TensorFlow ComplEx Neural link prediction model by Trouillon et al. (2016)

Tim Rockt¨ aschel GPU-accelerated End-to-end Differentiable Planning and Reasoning 28/39

Details

Models implemented in TensorFlow ComplEx Neural link prediction model by Trouillon et al. (2016) NTP End-to-end differentiable prover

Tim Rockt¨ aschel GPU-accelerated End-to-end Differentiable Planning and Reasoning 28/39

Details

Models implemented in TensorFlow ComplEx Neural link prediction model by Trouillon et al. (2016) NTP End-to-end differentiable prover NTPλ Prover trained with ComplEx as auxiliary loss

Tim Rockt¨ aschel GPU-accelerated End-to-end Differentiable Planning and Reasoning 28/39

Details

Models implemented in TensorFlow ComplEx Neural link prediction model by Trouillon et al. (2016) NTP End-to-end differentiable prover NTPλ Prover trained with ComplEx as auxiliary loss Rule Templates:

Kinship, Nations & UMLS 20 #1(X, Y) :– #2(X, Y). 20 #1(X, Y) :– #2(Y, X). 20 #1(X, Y) :– #2(X, Z), #3(Z, Y). Countries S1 3 #1(X, Y) :– #1(Y, X). 3 #1(X, Y) :– #2(X, Z), #2(Z, Y). Countries S2 3 #1(X, Y) :– #2(X, Z), #3(Z, Y). Countries S3 3 #1(X, Y) :– #2(X, Z), #3(Z, W), #4(W, Y). Tim Rockt¨ aschel GPU-accelerated End-to-end Differentiable Planning and Reasoning 28/39

Tim Rockt¨ aschel GPU-accelerated End-to-end Differentiable Planning and Reasoning 29/39

Results

Corpus Metric Model ComplEx NTP NTPλ Countries S1 AUC-PR 99.37 ± 0.4 90.83 ± 15.4 100.00 ± 0.0 S2 AUC-PR 87.95 ± 2.8 87.40 ± 11.7 93.04 ± 0.4 S3 AUC-PR 48.44 ± 6.3 56.68 ± 17.6 77.26 ± 17.0 Kinship MRR 0.46 0.36 0.48 HITS@1 0.34 0.24 0.39 HITS@3 0.49 0.40 0.47 HITS@10 0.74 0.60 0.71 Nations MRR 0.60 0.63 0.62 HITS@1 0.46 0.48 0.45 HITS@3 0.67 0.69 0.72 HITS@10 0.97 0.98 0.99 UMLS MRR 0.58 0.57 0.60 HITS@1 0.47 0.47 0.51 HITS@3 0.63 0.60 0.64 HITS@10 0.80 0.79 0.81

Tim Rockt¨ aschel GPU-accelerated End-to-end Differentiable Planning and Reasoning 30/39

Results

Corpus Examples of induced rules and their confidence Countries S1 0.90 locatedIn(X,Y) :– locatedIn(X,Z), locatedIn(Z,Y). S2 0.63 locatedIn(X,Y) :– neighborOf(X,Z), locatedIn(Z,Y). S3 0.32 locatedIn(X,Y) :– neighborOf(X,Z), neighborOf(Z,W), locatedIn(W,Y). Nations 0.68 blockpositionindex(X,Y) :– blockpositionindex(Y,X). 0.46 expeldiplomats(X,Y) :– negativebehavior(X,Y). 0.38 negativecomm(X,Y) :– commonbloc0(X,Y). 0.38 intergovorgs3(X,Y) :– intergovorgs(Y,X). UMLS 0.88 interacts with(X,Y) :– interacts with(X,Z), interacts with(Z,Y). 0.77 isa(X,Y) :– isa(X,Z), isa(Z,Y). 0.71 derivative of(X,Y) :– derivative of(X,Z), derivative of(Z,Y). Tim Rockt¨ aschel GPU-accelerated End-to-end Differentiable Planning and Reasoning 31/39

End-to-end Differentiable Planning

work-in-progress

Tim Rockt¨ aschel GPU-accelerated End-to-end Differentiable Planning and Reasoning 32/39

DQN

Mnih et al. (2015) 33/39

DQN zt

• t

encode

Q

evaluate

Mnih et al. (2015) 33/39

DQN zt

• t

encode

Q

evaluate

L(θ) =

  

target

• r
• reward

+ γ

• discount

max

a′ Q(s′, a′, θ−) −Q(s, a, θ)

  

2 Mnih et al. (2015) 33/39

Tim Rockt¨ aschel GPU-accelerated End-to-end Differentiable Planning and Reasoning 34/39

Tim Rockt¨ aschel GPU-accelerated End-to-end Differentiable Planning and Reasoning 35/39

Tree Planning

zt

• t

encode Tim Rockt¨ aschel GPU-accelerated End-to-end Differentiable Planning and Reasoning 36/39

Tree Planning

zt

• t

encode

a1 a2 a3

transition Tim Rockt¨ aschel GPU-accelerated End-to-end Differentiable Planning and Reasoning 36/39

Tree Planning

Tree Transitioning zt

• t

encode

a1 a2 a3

transition transition transition transition Tim Rockt¨ aschel GPU-accelerated End-to-end Differentiable Planning and Reasoning 36/39

Tree Planning

Tree Transitioning zt

• t

encode

a1 a2 a3

transition transition transition transition evaluate evaluate evaluate Tim Rockt¨ aschel GPU-accelerated End-to-end Differentiable Planning and Reasoning 36/39

Tree Planning

Tree Transitioning zt

• t

encode

a1 a2 a3

transition transition transition transition Q(ot, a3) max evaluate Q(ot, a2) max evaluate Q(ot, a1) max evaluate Tim Rockt¨ aschel GPU-accelerated End-to-end Differentiable Planning and Reasoning 36/39

Tree Planning

Tree Transitioning Value Prediction zt

• t

encode

a1 a2 a3

transition transition transition transition

Q

Q(ot, a3) max evaluate Q(ot, a2) max evaluate Q(ot, a1) max evaluate Tim Rockt¨ aschel GPU-accelerated End-to-end Differentiable Planning and Reasoning 36/39

Results

0.0 0.5 1.0 1.5 2.0 2.5 3.0 3.5 4.0 1e7 200 400 600 800 1000

DQN 938 TreeQN 1028

Enduro Steps Average Reward over 100 Episodes

Tim Rockt¨ aschel GPU-accelerated End-to-end Differentiable Planning and Reasoning 37/39

Results

0.0 0.5 1.0 1.5 2.0 2.5 3.0 3.5 4.0 1e7 500 1000 1500 2000 2500 3000

DQN 2497 TreeQN 3467

Alien Steps Average Reward over 100 Episodes

Tim Rockt¨ aschel GPU-accelerated End-to-end Differentiable Planning and Reasoning 37/39

Results

0.0 0.5 1.0 1.5 2.0 2.5 3.0 3.5 4.0 1e7 1000 2000 3000 4000

DQN 3854 TreeQN 4670

MsPacman Steps Average Reward over 100 Episodes

Tim Rockt¨ aschel GPU-accelerated End-to-end Differentiable Planning and Reasoning 37/39

Results

0.0 0.5 1.0 1.5 2.0 2.5 3.0 3.5 4.0 1e7 2000 4000 6000 8000 10000 12000

DQN 7845 TreeQN 13856

Seaquest Steps Average Reward over 100 Episodes

Tim Rockt¨ aschel GPU-accelerated End-to-end Differentiable Planning and Reasoning 37/39

<

<

Summary

Prolog’s backward chaining as recipe for recursively constructing a neural network to prove facts in a knowledge base

Tim Rockt¨ aschel GPU-accelerated End-to-end Differentiable Planning and Reasoning 39/39

Summary

Prolog’s backward chaining as recipe for recursively constructing a neural network to prove facts in a knowledge base Proof success differentiable w.r.t. subsymbolic representations

Tim Rockt¨ aschel GPU-accelerated End-to-end Differentiable Planning and Reasoning 39/39

Summary

Prolog’s backward chaining as recipe for recursively constructing a neural network to prove facts in a knowledge base Proof success differentiable w.r.t. subsymbolic representations Can learn vector representations of symbols and induce interpretable rules

• f predefined structure

Tim Rockt¨ aschel GPU-accelerated End-to-end Differentiable Planning and Reasoning 39/39

Summary

Prolog’s backward chaining as recipe for recursively constructing a neural network to prove facts in a knowledge base Proof success differentiable w.r.t. subsymbolic representations Can learn vector representations of symbols and induce interpretable rules

• f predefined structure

Various GPU optimizations: batch proving, tree pruning etc.

Tim Rockt¨ aschel GPU-accelerated End-to-end Differentiable Planning and Reasoning 39/39

Summary

Prolog’s backward chaining as recipe for recursively constructing a neural network to prove facts in a knowledge base Proof success differentiable w.r.t. subsymbolic representations Can learn vector representations of symbols and induce interpretable rules

• f predefined structure

Various GPU optimizations: batch proving, tree pruning etc. Outperform neural link prediction model on benchmark knowledge bases

Tim Rockt¨ aschel GPU-accelerated End-to-end Differentiable Planning and Reasoning 39/39

Summary

Prolog’s backward chaining as recipe for recursively constructing a neural network to prove facts in a knowledge base Proof success differentiable w.r.t. subsymbolic representations Can learn vector representations of symbols and induce interpretable rules

• f predefined structure

Various GPU optimizations: batch proving, tree pruning etc. Outperform neural link prediction model on benchmark knowledge bases Future research:

Tim Rockt¨ aschel GPU-accelerated End-to-end Differentiable Planning and Reasoning 39/39

Summary

Prolog’s backward chaining as recipe for recursively constructing a neural network to prove facts in a knowledge base Proof success differentiable w.r.t. subsymbolic representations Can learn vector representations of symbols and induce interpretable rules

• f predefined structure

Various GPU optimizations: batch proving, tree pruning etc. Outperform neural link prediction model on benchmark knowledge bases Future research:

Scale to larger knowledge bases

Tim Rockt¨ aschel GPU-accelerated End-to-end Differentiable Planning and Reasoning 39/39

Summary

Prolog’s backward chaining as recipe for recursively constructing a neural network to prove facts in a knowledge base Proof success differentiable w.r.t. subsymbolic representations Can learn vector representations of symbols and induce interpretable rules

• f predefined structure

Various GPU optimizations: batch proving, tree pruning etc. Outperform neural link prediction model on benchmark knowledge bases Future research:

Scale to larger knowledge bases Connect to RNNs for natural language statements

Tim Rockt¨ aschel GPU-accelerated End-to-end Differentiable Planning and Reasoning 39/39

Summary

Prolog’s backward chaining as recipe for recursively constructing a neural network to prove facts in a knowledge base Proof success differentiable w.r.t. subsymbolic representations Can learn vector representations of symbols and induce interpretable rules

• f predefined structure

Various GPU optimizations: batch proving, tree pruning etc. Outperform neural link prediction model on benchmark knowledge bases Future research:

Scale to larger knowledge bases Connect to RNNs for natural language statements Proving of mathematical theorems

Tim Rockt¨ aschel GPU-accelerated End-to-end Differentiable Planning and Reasoning 39/39

Summary

Prolog’s backward chaining as recipe for recursively constructing a neural network to prove facts in a knowledge base Proof success differentiable w.r.t. subsymbolic representations Can learn vector representations of symbols and induce interpretable rules

• f predefined structure

Various GPU optimizations: batch proving, tree pruning etc. Outperform neural link prediction model on benchmark knowledge bases Future research:

Scale to larger knowledge bases Connect to RNNs for natural language statements Proving of mathematical theorems Visual reasoning

Tim Rockt¨ aschel GPU-accelerated End-to-end Differentiable Planning and Reasoning 39/39

Summary

Prolog’s backward chaining as recipe for recursively constructing a neural network to prove facts in a knowledge base Proof success differentiable w.r.t. subsymbolic representations Can learn vector representations of symbols and induce interpretable rules

• f predefined structure

Various GPU optimizations: batch proving, tree pruning etc. Outperform neural link prediction model on benchmark knowledge bases Future research:

Scale to larger knowledge bases Connect to RNNs for natural language statements Proving of mathematical theorems Visual reasoning

Encouraging preliminary results using tree planning for Atari

Tim Rockt¨ aschel GPU-accelerated End-to-end Differentiable Planning and Reasoning 39/39

Thank you!

http://rockt.github.com tim.rocktaschel@cs.ox.ac.uk Twitter: @ rockt

References

Guillaume Bouchard, Sameer Singh, and Theo Trouillon. 2015. On approximate reasoning capabilities of low-rank vector spaces. In Proceedings of the 2015 AAAI Spring Symposium on Knowledge Representation and Reasoning (KRR): Integrating Symbolic and Neural Approaches. Rajarshi Das, Arvind Neelakantan, David Belanger, and Andrew McCallum. 2017. Chains of reasoning over entities, relations, and text using recurrent neural networks. In Conference of the European Chapter of the Association for Computational Linguistics (EACL). Stanley Kok and Pedro M. Domingos. 2007. Statistical predicate invention. In Machine Learning, Proceedings of the Twenty-Fourth International Conference (ICML 2007), Corvallis, Oregon, USA, June 20-24, 2007, pages 433–440. Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A. Rusu, Joel Veness, Marc G. Bellemare, Alex Graves, Martin A. Riedmiller, Andreas Fidjeland, Georg Ostrovski, Stig Petersen, Charles Beattie, Amir Sadik, Ioannis Antonoglou, Helen King, Dharshan Kumaran, Daan Wierstra, Shane Legg, and Demis Hassabis. 2015. Human-level control through deep reinforcement learning. Nature, 518(7540):529–533. URL https://doi.org/10.1038/nature14236. Tim Rockt¨ aschel and Sebastian Riedel. 2017. End-to-end differentiable proving. In Advances in Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, California, United States, volume abs/1705.11040. Th´ eo Trouillon, Johannes Welbl, Sebastian Riedel, ´ Eric Gaussier, and Guillaume Bouchard. 2016. Complex embeddings for simple link

• prediction. In Proceedings of the 33nd International Conference on Machine Learning, ICML 2016, New York City, NY, USA, June

19-24, 2016, pages 2071–2080. Bishan Yang, Wen-tau Yih, Xiaodong He, Jianfeng Gao, and Li Deng. 2015. Embedding entities and relations for learning and inference in knowledge bases. In International Conference on Learning Representations (ICLR).