Learning and Reasoning in Logic Tensor Networks Luciano Serafini 1 , - - PowerPoint PPT Presentation

▶
learning and reasoning in logic tensor networks
SMART_READER_LITE
LIVE PREVIEW

Learning and Reasoning in Logic Tensor Networks Luciano Serafini 1 , - - PowerPoint PPT Presentation

Jun 06, 2023 425 likes •819 views

Learning and Reasoning in Logic Tensor Networks Luciano Serafini 1 , Ivan Donadello 1 , 2 , Artur dAvila Garces 3 1 Fondazione Bruno Kessler, Italy 2 University of Trento, Italy 3 City University London, UK May 7, 2017 Luciano Serafini, Ivan

slide-1
SLIDE 1

Learning and Reasoning in Logic Tensor Networks

Luciano Serafini1, Ivan Donadello1,2, Artur d’Avila Garces3

1Fondazione Bruno Kessler, Italy 2University of Trento, Italy 3City University London, UK

May 7, 2017

Luciano Serafini, Ivan Donadello, Artur d’Avila Garces ( Fondazione Bruno Kessler, Italy University of Trento, Italy City University London, Learning and Reasoning in Logic Tensor Networks May 7, 2017 1 / 23

slide-2
SLIDE 2

The SRL Mindmap

AI

KRR

SRL

Learning Planning NLP Perception . . .

Statistical Relational Learning

is a subdiscipline of artificial intelligence that is concerned with domain models that exhibit both uncertainty and complex relational structure.

Luciano Serafini, Ivan Donadello, Artur d’Avila Garces ( Fondazione Bruno Kessler, Italy University of Trento, Italy City University London, Learning and Reasoning in Logic Tensor Networks May 7, 2017 2 / 23

slide-3
SLIDE 3

Hybrid domains

We are interested in Statistical Relational Learning over hybrid domains, i.e., domains that are characterized by the presence of structured data (categorical/semantic); continuous data (continuous features);

Luciano Serafini, Ivan Donadello, Artur d’Avila Garces ( Fondazione Bruno Kessler, Italy University of Trento, Italy City University London, Learning and Reasoning in Logic Tensor Networks May 7, 2017 3 / 23

slide-4
SLIDE 4

Hybrid domains

Example (SRL domain)

Kurt person Car2 car Rome town FCA company Detroit town 10000 dollar 15342 dollar 130.00 hp 53.72 km2 34 years

  • wns

livesIn madeBy locatedIn price engine power income age area 2/2/95 date since

Luciano Serafini, Ivan Donadello, Artur d’Avila Garces ( Fondazione Bruno Kessler, Italy University of Trento, Italy City University London, Learning and Reasoning in Logic Tensor Networks May 7, 2017 4 / 23

slide-5
SLIDE 5

Tasks in Statistical Relational Learning

Object Classification: Predicting the type of an

  • bject based on its relations

and attributes; Reletion detenction: Predicting if two objects are connected by a relation, based

  • n types and attributes of the

participating objects; Regression: predicting the (distribution of) values of the attributies of an object, (a pair of related objects) based

  • n the types and relations of

the object(s) involved.

Example (SRL domain)

Kurt person Car2 car Rome town FCA company Detroit town 10000 dollar 15342 dollar 130.00 hp 53.72 km2 34 years

  • wns

livesIn madeBy locatedIn price engine power income age area 2/2/95 date since

Luciano Serafini, Ivan Donadello, Artur d’Avila Garces ( Fondazione Bruno Kessler, Italy University of Trento, Italy City University London, Learning and Reasoning in Logic Tensor Networks May 7, 2017 5 / 23

slide-6
SLIDE 6

Real-world uncertain, structured and hybrid domains

Robotics: a robot’s location is a continuous values while the the types of the objects it encounters can be described by discrete set

  • f classes

Semantic Image Interpretation: The visual features of a bounding box of a picture are con- tinuous values, while the types of objects con- tained in a bounding box and the relations be- tween them are taken from a discrete set Natural Language Processing: The distri- butional semantics provide a vectorial (numer- ical) representation of the meaning of words, while WordNet associates to each word a set of synsets and a set of relations with other words which are finite and discrete

Luciano Serafini, Ivan Donadello, Artur d’Avila Garces ( Fondazione Bruno Kessler, Italy University of Trento, Italy City University London, Learning and Reasoning in Logic Tensor Networks May 7, 2017 6 / 23

slide-7
SLIDE 7

Semantic Image interpretation

semantic Image Interpretation (SII)

Luciano Serafini, Ivan Donadello, Artur d’Avila Garces ( Fondazione Bruno Kessler, Italy University of Trento, Italy City University London, Learning and Reasoning in Logic Tensor Networks May 7, 2017 7 / 23

slide-8
SLIDE 8

Semantic Image interpretation

semantic Image Interpretation (SII)

detect the main objects shown in the picture;

Luciano Serafini, Ivan Donadello, Artur d’Avila Garces ( Fondazione Bruno Kessler, Italy University of Trento, Italy City University London, Learning and Reasoning in Logic Tensor Networks May 7, 2017 7 / 23

slide-9
SLIDE 9

Semantic Image interpretation

semantic Image Interpretation (SII)

detect the main objects shown in the picture; assign to each object an object type;

ball player leg player leg leg number logo

Luciano Serafini, Ivan Donadello, Artur d’Avila Garces ( Fondazione Bruno Kessler, Italy University of Trento, Italy City University London, Learning and Reasoning in Logic Tensor Networks May 7, 2017 7 / 23

slide-10
SLIDE 10

Semantic Image interpretation

semantic Image Interpretation (SII)

detect the main objects shown in the picture; assign to each object an object type; determine the relations between the objects as shown in the picture represent the outcome of the detection in a semantic structure.

ball player leg player leg leg number logo b1 ball b2 player b3 player b4 leg b5 leg b6 leg b7 number b8 logo kicks partOf partOf hasNum attaks

Luciano Serafini, Ivan Donadello, Artur d’Avila Garces ( Fondazione Bruno Kessler, Italy University of Trento, Italy City University London, Learning and Reasoning in Logic Tensor Networks May 7, 2017 7 / 23

slide-11
SLIDE 11

Language - to specify knowledge about models

Two sorted first order language: (abstract sort and numeric sort) Abstract constant symbols (b1,b2,. . . ,b8); Abstract relation symbols (player(x), ball(x), partOf(x,y),hasNum(x,y); Numeric function symbols (xBL(x),yBL(x),width(x),height(h) area(x),color(x),contRatio(x,y); COLOR CODE: denotes objects and relations of the domain structure; denotes attributes and relations between attributes of the numeric part of the domain.

Luciano Serafini, Ivan Donadello, Artur d’Avila Garces ( Fondazione Bruno Kessler, Italy University of Trento, Italy City University London, Learning and Reasoning in Logic Tensor Networks May 7, 2017 8 / 23

slide-12
SLIDE 12

Domain description and queries

Example (Domain descritpion:)

knowledge about object detection: xBL(b1) = 23, yBL(b1) = 73, width(b1) = 20, height(b1) = 21 xBL(b2) = 45, yBL(b1) = 70, width(b1) = 40, height(b1) = 104 . . . contRatio(b2, b4) = 1.0, contRatio(b2, b5) = 0.4, . . .

Luciano Serafini, Ivan Donadello, Artur d’Avila Garces ( Fondazione Bruno Kessler, Italy University of Trento, Italy City University London, Learning and Reasoning in Logic Tensor Networks May 7, 2017 9 / 23

slide-13
SLIDE 13

Domain description and queries

Example (Domain descritpion:)

knowledge about object detection: xBL(b1) = 23, yBL(b1) = 73, width(b1) = 20, height(b1) = 21 xBL(b2) = 45, yBL(b1) = 70, width(b1) = 40, height(b1) = 104 . . . contRatio(b2, b4) = 1.0, contRatio(b2, b5) = 0.4, . . . partial knowledge about object types and relations ball(b1), player(b2), player(b3), leg(b4), leg(b5), partOf (b3, b2), kicks(b2, b1), hasNum(b3, b7),. . .

Luciano Serafini, Ivan Donadello, Artur d’Avila Garces ( Fondazione Bruno Kessler, Italy University of Trento, Italy City University London, Learning and Reasoning in Logic Tensor Networks May 7, 2017 9 / 23

slide-14
SLIDE 14

Domain description and queries

Example (Domain descritpion:)

knowledge about object detection: xBL(b1) = 23, yBL(b1) = 73, width(b1) = 20, height(b1) = 21 xBL(b2) = 45, yBL(b1) = 70, width(b1) = 40, height(b1) = 104 . . . contRatio(b2, b4) = 1.0, contRatio(b2, b5) = 0.4, . . . partial knowledge about object types and relations ball(b1), player(b2), player(b3), leg(b4), leg(b5), partOf (b3, b2), kicks(b2, b1), hasNum(b3, b7),. . .

  • ntological axioms

∀xy.partOf (x, y) ∧ leg(x) → player(y), ∀xy, kick(x, y) → player(x) ∧ ball(y), ∀xypartOf (x, y) → contRatio(x, y) > .9 ∀xplayer(x) → ¬ball(x),

Luciano Serafini, Ivan Donadello, Artur d’Avila Garces ( Fondazione Bruno Kessler, Italy University of Trento, Italy City University London, Learning and Reasoning in Logic Tensor Networks May 7, 2017 9 / 23

slide-15
SLIDE 15

Domain description and queries

Example (Domain descritpion:)

knowledge about object detection: xBL(b1) = 23, yBL(b1) = 73, width(b1) = 20, height(b1) = 21 xBL(b2) = 45, yBL(b1) = 70, width(b1) = 40, height(b1) = 104 . . . contRatio(b2, b4) = 1.0, contRatio(b2, b5) = 0.4, . . . partial knowledge about object types and relations ball(b1), player(b2), player(b3), leg(b4), leg(b5), partOf (b3, b2), kicks(b2, b1), hasNum(b3, b7),. . .

  • ntological axioms

∀xy.partOf (x, y) ∧ leg(x) → player(y), ∀xy, kick(x, y) → player(x) ∧ ball(y), ∀xypartOf (x, y) → contRatio(x, y) > .9 ∀xplayer(x) → ¬ball(x),

Example (Queries)

Query about missing knowledge about

  • bject tipes and relations

player(b10)

  • xBL(b10) = 83,

yBL(b10 = 42, width(b10 = 30 . . . partOf (b10, b11)

  • xBL(b10) = 83,

yBL(b10 = 42, width(b10 = 30 . . . xBL(b11) = 83, yBL(b11 = 42, width(b11 = 30 . . . contRatio(b10, b11) = 0.6 contRatio(b11, b10) = 0.9 . . .

Luciano Serafini, Ivan Donadello, Artur d’Avila Garces ( Fondazione Bruno Kessler, Italy University of Trento, Italy City University London, Learning and Reasoning in Logic Tensor Networks May 7, 2017 9 / 23

slide-16
SLIDE 16

Logic Tensor Network basic idea

f (x) f (y) g(x) g(y) h(x, y) h(y, x) Logic Tensor Network that computes the truth value of the formula φ(x, y) on the basis of the numeric features

  • f x, y and the pair x, y

φ(x, y)

Luciano Serafini, Ivan Donadello, Artur d’Avila Garces ( Fondazione Bruno Kessler, Italy University of Trento, Italy City University London, Learning and Reasoning in Logic Tensor Networks May 7, 2017 10 / 23

slide-17
SLIDE 17

Logic Tensor Network basic idea

f (x) f (y) g(x) g(y) h(x, y) h(y, x) Logic Tensor Network that computes the truth value of the formula φ(x, y) on the basis of the numeric features

  • f x, y and the pair x, y

φ(x, y) Deep Neural networks that compute the values of all the atomic formulas compos- ing φ(x, y) starting from the numeric features Netrork for fuzzy logic

Luciano Serafini, Ivan Donadello, Artur d’Avila Garces ( Fondazione Bruno Kessler, Italy University of Trento, Italy City University London, Learning and Reasoning in Logic Tensor Networks May 7, 2017 10 / 23

slide-18
SLIDE 18

LTN for predicates

n unary numeric function f1(x), . . . , fn(x) and m binary numeric function g1(x, y), . . . , gm(x, y)

LTN for unary predicate/type P(x)

LTNP(v) = σ

  • u⊺

P tanh

  • v⊺W [1:k]

P

v + VPv + bP

  • wP ∈ Rk×n×n, VP ∈ Rk×n, bP ∈ Rk, and uP ∈ Rk are parameters.

LTN for binary relation R(x, y)

LTNP(v) = σ

  • u⊺

P tanh

  • v⊺W [1:k]

P

v + VPv + bP

  • wP ∈ Rk×h×h, VP ∈ Rk×h, bP ∈ Rk, and uP ∈ Rk are parameters, and

h = 2(n + m) = the total number of numeric features that can be

  • btained applying fi and gi to x and y.

Luciano Serafini, Ivan Donadello, Artur d’Avila Garces ( Fondazione Bruno Kessler, Italy University of Trento, Italy City University London, Learning and Reasoning in Logic Tensor Networks May 7, 2017 11 / 23

slide-19
SLIDE 19

Fuzzy semantics for propositional connectives

In fuzzy semantics atoms are assigned with some truth value in real interval [0,1] connectives have functional semantics. e.g., a binary connective ◦ must be interpreted in a function f◦ : [0, 1]2 → [0, 1]. Truth values are ordeblue, i.e., if x > y, then x is a stronger truth than y Generalization of classical propositional logic: 0 corresponds to FALSE and 1 corresponds to TRUE

Luciano Serafini, Ivan Donadello, Artur d’Avila Garces ( Fondazione Bruno Kessler, Italy University of Trento, Italy City University London, Learning and Reasoning in Logic Tensor Networks May 7, 2017 12 / 23

slide-20
SLIDE 20

Fuzzy semantics for connectives and quantifiers

Lukasiewicz T-norm, T-conorm, residual, and precomplement

T-norm a ∧ b = max(0, a + b − 1) T-conorm a ∨ b = min(1, a + b) residual a → = if a > b 1 − a + b if a ≤ b 1 precomplement ¬a = 1 − a aggregation ∀x.a(x) = limn→∞ 1

n

n

i=1(a(i)−1−1

Alternatively, use G¨

  • del or Product T-norm, and geometric or aritmetic

mean as aggregator.

Luciano Serafini, Ivan Donadello, Artur d’Avila Garces ( Fondazione Bruno Kessler, Italy University of Trento, Italy City University London, Learning and Reasoning in Logic Tensor Networks May 7, 2017 13 / 23

slide-21
SLIDE 21

Constructive semantics for Existential quantifier

LTN interprets existential quantifiers constructively via Skolemization. Every formula ∀x1, . . . , xn∃yφ(x1, . . . , xn, y) is rewritten as ∀x1, . . . , xmφ(x1, . . . , xn, f (x1, . . . , xm)), by introducing a new m-ary function symbol f ,

Example

∀x.(cat(x) → ∃y.partof (y, x) ∧ tail(y)) is transformed in ∀x(cat(x) → partOf (tailOf (x), x) ∧ tail(tailOf (x)))

Luciano Serafini, Ivan Donadello, Artur d’Avila Garces ( Fondazione Bruno Kessler, Italy University of Trento, Italy City University London, Learning and Reasoning in Logic Tensor Networks May 7, 2017 14 / 23

slide-22
SLIDE 22

Grounding = relation between logical symbols and data

v = v1, . . . , vn u = u1, . . . , un W 1

P

W 2

P

V 1

P

V 2

P

B1

P

B2

P

+ + th th uP 1 − σ W 1

A

W 2

A

V 1

A

V 2

A

B1

A

B2

A

+ + th th uA σ max G(P(v, u) → A(u) Luciano Serafini, Ivan Donadello, Artur d’Avila Garces ( Fondazione Bruno Kessler, Italy University of Trento, Italy City University London, Learning and Reasoning in Logic Tensor Networks May 7, 2017 15 / 23

slide-23
SLIDE 23

Grounding = relation between logical symbols and data

G(¬P(v, u)) G(A(u)) v = v1, . . . , vn u = u1, . . . , un W 1

P

W 2

P

V 1

P

V 2

P

B1

P

B2

P

+ + th th uP 1 − σ W 1

A

W 2

A

V 1

A

V 2

A

B1

A

B2

A

+ + th th uA σ max G(P(v, u) → A(u) Luciano Serafini, Ivan Donadello, Artur d’Avila Garces ( Fondazione Bruno Kessler, Italy University of Trento, Italy City University London, Learning and Reasoning in Logic Tensor Networks May 7, 2017 15 / 23

slide-24
SLIDE 24

Parameter learning = best satisfiability

Given a FOL theory K the best satisfiability problem as the problem of finding the set of parameters Θ of the LTN, then the problems become G∗ = LTN(K, Θ∗) Θ∗ = argmax

Θ

  • min

K| =φ LTN(K, Θ)(φ)

  • Luciano Serafini, Ivan Donadello, Artur d’Avila Garces ( Fondazione Bruno Kessler, Italy University of Trento, Italy City University London,

Learning and Reasoning in Logic Tensor Networks May 7, 2017 16 / 23

slide-25
SLIDE 25

Learning from model description and answering queries

xBL(b1) = 23, yBL(b1) = 73, width(b1) = 20, height(b1) = 21 xBL(b2) = 45, yBL(b1) = 70, width(b1) = 40, height(b1) = 104 . . . contRatio(b2, b4) = 1.0, contRatio(b2, b5) = 0.4, . . . ball(b1), player(b2), player(b3), leg(b4), leg(b5), partOf (b3, b2), kicks(b2, b1), hasNum(b3, b7),. . . ∀xy.partOf (x, y) ∧ leg(x) → player(y), ∀xy, kick(x, y) → player(x) ∧ ball(y), ∀xypartOf (x, y) → contRatio(x, y) > .9 ∀xplayer(x) → ¬ball(x), K

Luciano Serafini, Ivan Donadello, Artur d’Avila Garces ( Fondazione Bruno Kessler, Italy University of Trento, Italy City University London, Learning and Reasoning in Logic Tensor Networks May 7, 2017 17 / 23

slide-26
SLIDE 26

Learning from model description and answering queries

Θ∗ = argmaxΘ

  • minK|

=φ LTN(K, Θ)(φ)

  • xBL(b1) = 23, yBL(b1) = 73,

width(b1) = 20, height(b1) = 21 xBL(b2) = 45, yBL(b1) = 70, width(b1) = 40, height(b1) = 104 . . . contRatio(b2, b4) = 1.0, contRatio(b2, b5) = 0.4, . . . ball(b1), player(b2), player(b3), leg(b4), leg(b5), partOf (b3, b2), kicks(b2, b1), hasNum(b3, b7),. . . ∀xy.partOf (x, y) ∧ leg(x) → player(y), ∀xy, kick(x, y) → player(x) ∧ ball(y), ∀xypartOf (x, y) → contRatio(x, y) > .9 ∀xplayer(x) → ¬ball(x), K

Luciano Serafini, Ivan Donadello, Artur d’Avila Garces ( Fondazione Bruno Kessler, Italy University of Trento, Italy City University London, Learning and Reasoning in Logic Tensor Networks May 7, 2017 17 / 23

slide-27
SLIDE 27

Learning from model description and answering queries

Θ∗ = argmaxΘ

  • minK|

=φ LTN(K, Θ)(φ)

  • xBL(b1) = 23, yBL(b1) = 73,

width(b1) = 20, height(b1) = 21 xBL(b2) = 45, yBL(b1) = 70, width(b1) = 40, height(b1) = 104 . . . contRatio(b2, b4) = 1.0, contRatio(b2, b5) = 0.4, . . . ball(b1), player(b2), player(b3), leg(b4), leg(b5), partOf (b3, b2), kicks(b2, b1), hasNum(b3, b7),. . . ∀xy.partOf (x, y) ∧ leg(x) → player(y), ∀xy, kick(x, y) → player(x) ∧ ball(y), ∀xypartOf (x, y) → contRatio(x, y) > .9 ∀xplayer(x) → ¬ball(x), K LTNK,Θ∗    player(b10)

  • xBL(b10) = 83,

yBL(b10 = 42, width(b10) = 30 . . .     Q

Luciano Serafini, Ivan Donadello, Artur d’Avila Garces ( Fondazione Bruno Kessler, Italy University of Trento, Italy City University London, Learning and Reasoning in Logic Tensor Networks May 7, 2017 17 / 23

slide-28
SLIDE 28

Semantic Image interpretation

semantic Image Interpretation (SII)

Luciano Serafini, Ivan Donadello, Artur d’Avila Garces ( Fondazione Bruno Kessler, Italy University of Trento, Italy City University London, Learning and Reasoning in Logic Tensor Networks May 7, 2017 18 / 23

slide-29
SLIDE 29

Semantic Image interpretation

semantic Image Interpretation (SII)

  • bject detection: Fast RCNN (state of the art object detector)

Luciano Serafini, Ivan Donadello, Artur d’Avila Garces ( Fondazione Bruno Kessler, Italy University of Trento, Italy City University London, Learning and Reasoning in Logic Tensor Networks May 7, 2017 18 / 23

slide-30
SLIDE 30

Semantic Image interpretation

semantic Image Interpretation (SII)

  • bject detection: Fast RCNN (state of the art object detector)

Fast-RCNN returns candidate bounding boxes, associated with weights for each object class;

xBL(b1) = 12 yBL(b1) = 27 width(b1) = 30 height(b1) = 30 rcnnball(b1) = .8 rcnnplayer(b1) = .3 rcnnlogo(b1) = .02 . . .

Luciano Serafini, Ivan Donadello, Artur d’Avila Garces ( Fondazione Bruno Kessler, Italy University of Trento, Italy City University London, Learning and Reasoning in Logic Tensor Networks May 7, 2017 18 / 23

slide-31
SLIDE 31

Semantic Image interpretation

semantic Image Interpretation (SII)

  • bject detection: Fast RCNN (state of the art object detector)

Fast-RCNN returns candidate bounding boxes, associated with weights for each object class;

xBL(b1) = 12 yBL(b1) = 27 width(b1) = 30 height(b1) = 30 rcnnball(b1) = .8 rcnnplayer(b1) = .3 rcnnlogo(b1) = .02 . . . xBL(b1) = 14 yBL(b1) = 17 width(b1) = 40 height(b1) = 100 rcnnball(b1) = .1 rcnnplayer(b1) = .7 rcnnlogo(b1) = .02 . . .

Luciano Serafini, Ivan Donadello, Artur d’Avila Garces ( Fondazione Bruno Kessler, Italy University of Trento, Italy City University London, Learning and Reasoning in Logic Tensor Networks May 7, 2017 18 / 23

slide-32
SLIDE 32

Semantic Image interpretation

semantic Image Interpretation (SII)

  • bject detection: Fast RCNN (state of the art object detector)

Fast-RCNN returns candidate bounding boxes, associated with weights for each object class;

xBL(b1) = 12 yBL(b1) = 27 width(b1) = 30 height(b1) = 30 rcnnball(b1) = .8 rcnnplayer(b1) = .3 rcnnlogo(b1) = .02 . . . xBL(b1) = 14 yBL(b1) = 17 width(b1) = 40 height(b1) = 100 rcnnball(b1) = .1 rcnnplayer(b1) = .7 rcnnlogo(b1) = .02 . . . xBL(b1) = 34 yBL(b1) = 37 width(b1) = 44 height(b1) = 130 rcnnball(b1) = .1 rcnnplayer(b1) = .74 rcnnlogo(b1) = .1 . . .

Luciano Serafini, Ivan Donadello, Artur d’Avila Garces ( Fondazione Bruno Kessler, Italy University of Trento, Italy City University London, Learning and Reasoning in Logic Tensor Networks May 7, 2017 18 / 23

slide-33
SLIDE 33

Semantic Image interpretation

semantic Image Interpretation (SII)

  • bject detection: Fast RCNN (state of the art object detector)

Fast-RCNN returns candidate bounding boxes, associated with weights for each object class; For each pair of bounding boxe we compute additional binary feature that measure the mutual overlap between the two bounding boxes.

xBL(b1) = 12 yBL(b1) = 27 width(b1) = 30 height(b1) = 30 rcnnball(b1) = .8 rcnnplayer(b1) = .3 rcnnlogo(b1) = .02 . . . xBL(b1) = 14 yBL(b1) = 17 width(b1) = 40 height(b1) = 100 rcnnball(b1) = .1 rcnnplayer(b1) = .7 rcnnlogo(b1) = .02 . . . xBL(b1) = 34 yBL(b1) = 37 width(b1) = 44 height(b1) = 130 rcnnball(b1) = .1 rcnnplayer(b1) = .74 rcnnlogo(b1) = .1 . . . contRatio(b2, b3) = 0.3 contRatio(b3, b2) = 0.2

Luciano Serafini, Ivan Donadello, Artur d’Avila Garces ( Fondazione Bruno Kessler, Italy University of Trento, Italy City University London, Learning and Reasoning in Logic Tensor Networks May 7, 2017 18 / 23

slide-34
SLIDE 34

LTN evaluation on PascalPart dataset

PascalPart contains 10103 pictures annotated with a set of bounding boxes labelled with object types (60 classes among animals, vehicles, and indor objects) We train an LTN with the approx 2/3 pictures and test on 1/3. by including the following background knowledge

◮ positive/negative examples for object classes (from training set)

weel(bb1), car(bb2), ¬horse(bb2), ¬person(bb4)

◮ positive/negative examples for relations (we focus on parthood

relation). partOf (bb1, bb2), ¬partOf (bb2, bb3), . . . ,

◮ general axioms about parthood relation:

∀x.car(x) ∧ partof (y, y) → wheeel(y) ∨ mirror(y) ∨ door(y) ∨ . . . ,)

Luciano Serafini, Ivan Donadello, Artur d’Avila Garces ( Fondazione Bruno Kessler, Italy University of Trento, Italy City University London, Learning and Reasoning in Logic Tensor Networks May 7, 2017 19 / 23

slide-35
SLIDE 35

LTN for SII results

LTNprior is an LTN trained with positive and negative examples + general axioms about partOo relation LTNexpl is an LTN trained only with positive and negative examples of types and partOf FRCNN is the baseline proposal classification for types given by Fast-RCNN RBPOF is the baseline for partOf based on the naive criteria area containment ≥ threshold

Luciano Serafini, Ivan Donadello, Artur d’Avila Garces ( Fondazione Bruno Kessler, Italy University of Trento, Italy City University London, Learning and Reasoning in Logic Tensor Networks May 7, 2017 20 / 23

slide-36
SLIDE 36

Robustness w.r.t. noisy data

logical axioms improve the robustness of the system in presence of noise in the labels of training data. e artificially add an increasing amount of noise to the PascalPart-dataset training data, and we measure the degradation of the performance,

Luciano Serafini, Ivan Donadello, Artur d’Avila Garces ( Fondazione Bruno Kessler, Italy University of Trento, Italy City University London, Learning and Reasoning in Logic Tensor Networks May 7, 2017 21 / 23

slide-37
SLIDE 37

Conclusions

we introduce Logic Tensor Networks, a general framework for SRL that integrates fuzzy logical reasoning and machine learning based on neural networks; We apply LTN to the challanging problem of semantic image interpretation; We experimentally show that the usage of logic based background knowledge improves the performance of automatic classification based only on numeric features.

Luciano Serafini, Ivan Donadello, Artur d’Avila Garces ( Fondazione Bruno Kessler, Italy University of Trento, Italy City University London, Learning and Reasoning in Logic Tensor Networks May 7, 2017 22 / 23

slide-38
SLIDE 38

Thanks

Thanks for your attention

Luciano Serafini, Ivan Donadello, Artur d’Avila Garces ( Fondazione Bruno Kessler, Italy University of Trento, Italy City University London, Learning and Reasoning in Logic Tensor Networks May 7, 2017 23 / 23