Learning and Reasoning in Logic Tensor Networks Luciano Serafini 1 , - PowerPoint PPT Presentation

Learning and Reasoning in Logic Tensor Networks Luciano Serafini 1 , Ivan Donadello 1 , 2 , Artur d’Avila Garces 3 1 Fondazione Bruno Kessler, Italy 2 University of Trento, Italy 3 City University London, UK May 7, 2017 Luciano Serafini, Ivan Donadello, Artur d’Avila Garces ( Fondazione Bruno Kessler, Italy University of Trento, Italy City University London, Learning and Reasoning in Logic Tensor Networks May 7, 2017 1 / 23

The SRL Mindmap . . . AI Perception NLP Statistical Relational Learning Planning is a subdiscipline of artificial intelligence that is concerned Learning with domain models that exhibit KRR both uncertainty and complex relational structure . SRL Luciano Serafini, Ivan Donadello, Artur d’Avila Garces ( Fondazione Bruno Kessler, Italy University of Trento, Italy City University London, Learning and Reasoning in Logic Tensor Networks May 7, 2017 2 / 23

Hybrid domains We are interested in Statistical Relational Learning over hybrid domains , i.e., domains that are characterized by the presence of structured data (categorical/semantic); continuous data (continuous features); Luciano Serafini, Ivan Donadello, Artur d’Avila Garces ( Fondazione Bruno Kessler, Italy University of Trento, Italy City University London, Learning and Reasoning in Logic Tensor Networks May 7, 2017 3 / 23

Hybrid domains Example (SRL domain) hp years dollar dollar 34 15342 10000 130.00 age income price engine power person car Kurt Car2 owns date since 2/2/95 madeBy livesIn company FCA Rome town locatedIn area km 2 Detroit town 53.72 Luciano Serafini, Ivan Donadello, Artur d’Avila Garces ( Fondazione Bruno Kessler, Italy University of Trento, Italy City University London, Learning and Reasoning in Logic Tensor Networks May 7, 2017 4 / 23

Tasks in Statistical Relational Learning Object Classification: Example (SRL domain) Predicting the type of an hp years dollar dollar object based on its relations 34 15342 10000 130.00 and attributes; Reletion detenction: age income price engine power Predicting if two objects are connected by a relation, based person car Kurt Car2 owns date on types and attributes of the since 2/2/95 madeBy participating objects; livesIn Regression: predicting the company FCA Rome town (distribution of) values of the attributies of an object, (a locatedIn area pair of related objects) based on the types and relations of km 2 Detroit town 53.72 the object(s) involved. Luciano Serafini, Ivan Donadello, Artur d’Avila Garces ( Fondazione Bruno Kessler, Italy University of Trento, Italy City University London, Learning and Reasoning in Logic Tensor Networks May 7, 2017 5 / 23

Real-world uncertain, structured and hybrid domains Robotics: a robot’s location is a continuous values while the the types of the objects it encounters can be described by discrete set of classes Semantic Image Interpretation: The visual features of a bounding box of a picture are con- tinuous values, while the types of objects con- tained in a bounding box and the relations be- tween them are taken from a discrete set Natural Language Processing: The distri- butional semantics provide a vectorial (numer- ical) representation of the meaning of words, while WordNet associates to each word a set of synsets and a set of relations with other words which are finite and discrete Luciano Serafini, Ivan Donadello, Artur d’Avila Garces ( Fondazione Bruno Kessler, Italy University of Trento, Italy City University London, Learning and Reasoning in Logic Tensor Networks May 7, 2017 6 / 23

Semantic Image interpretation semantic Image Interpretation (SII) Luciano Serafini, Ivan Donadello, Artur d’Avila Garces ( Fondazione Bruno Kessler, Italy University of Trento, Italy City University London, Learning and Reasoning in Logic Tensor Networks May 7, 2017 7 / 23

Semantic Image interpretation semantic Image Interpretation (SII) detect the main objects shown in the picture; Luciano Serafini, Ivan Donadello, Artur d’Avila Garces ( Fondazione Bruno Kessler, Italy University of Trento, Italy City University London, Learning and Reasoning in Logic Tensor Networks May 7, 2017 7 / 23

Semantic Image interpretation semantic Image Interpretation (SII) detect the main objects shown in the picture; assign to each object an object type; logo number player ball player leg leg leg Luciano Serafini, Ivan Donadello, Artur d’Avila Garces ( Fondazione Bruno Kessler, Italy University of Trento, Italy City University London, Learning and Reasoning in Logic Tensor Networks May 7, 2017 7 / 23

Semantic Image interpretation semantic Image Interpretation (SII) detect the main objects shown in the picture; assign to each object an object type; determine the relations between the objects as shown in the picture represent the outcome of the detection in a semantic structure. number b 8 logo logo b 7 player ball hasNum number player ball player player b 1 b 2 b 3 kicks attaks leg leg partOf partOf leg leg b 5 leg b 6 leg b 4 Luciano Serafini, Ivan Donadello, Artur d’Avila Garces ( Fondazione Bruno Kessler, Italy University of Trento, Italy City University London, Learning and Reasoning in Logic Tensor Networks May 7, 2017 7 / 23

Language - to specify knowledge about models Two sorted first order language: (abstract sort and numeric sort) Abstract constant symbols ( b 1 , b 2 ,. . . , b 8 ); Abstract relation symbols (player(x), ball(x), partOf(x,y),hasNum(x,y); Numeric function symbols (xBL(x),yBL(x),width(x),height(h) area(x),color(x),contRatio(x,y); COLOR CODE: denotes objects and relations of the domain structure; denotes attributes and relations between attributes of the numeric part of the domain. Luciano Serafini, Ivan Donadello, Artur d’Avila Garces ( Fondazione Bruno Kessler, Italy University of Trento, Italy City University London, Learning and Reasoning in Logic Tensor Networks May 7, 2017 8 / 23

Domain description and queries Example (Domain descritpion:) knowledge about object detection: xBL ( b 1 ) = 23, yBL ( b 1 ) = 73, width ( b 1 ) = 20, height ( b 1 ) = 21 xBL ( b 2 ) = 45, yBL ( b 1 ) = 70, width ( b 1 ) = 40, height ( b 1 ) = 104 . . . contRatio ( b 2 , b 4 ) = 1 . 0, contRatio ( b 2 , b 5 ) = 0 . 4, . . . Luciano Serafini, Ivan Donadello, Artur d’Avila Garces ( Fondazione Bruno Kessler, Italy University of Trento, Italy City University London, Learning and Reasoning in Logic Tensor Networks May 7, 2017 9 / 23

Domain description and queries Example (Domain descritpion:) knowledge about object detection: xBL ( b 1 ) = 23, yBL ( b 1 ) = 73, width ( b 1 ) = 20, height ( b 1 ) = 21 xBL ( b 2 ) = 45, yBL ( b 1 ) = 70, width ( b 1 ) = 40, height ( b 1 ) = 104 . . . contRatio ( b 2 , b 4 ) = 1 . 0, contRatio ( b 2 , b 5 ) = 0 . 4, . . . partial knowledge about object types and relations ball ( b 1 ), player ( b 2 ), player ( b 3 ), leg ( b 4 ), leg ( b 5 ), partOf ( b 3 , b 2 ), kicks ( b 2 , b 1 ), hasNum ( b 3 , b 7 ),. . . Luciano Serafini, Ivan Donadello, Artur d’Avila Garces ( Fondazione Bruno Kessler, Italy University of Trento, Italy City University London, Learning and Reasoning in Logic Tensor Networks May 7, 2017 9 / 23

Domain description and queries Example (Domain descritpion:) knowledge about object detection: xBL ( b 1 ) = 23, yBL ( b 1 ) = 73, width ( b 1 ) = 20, height ( b 1 ) = 21 xBL ( b 2 ) = 45, yBL ( b 1 ) = 70, width ( b 1 ) = 40, height ( b 1 ) = 104 . . . contRatio ( b 2 , b 4 ) = 1 . 0, contRatio ( b 2 , b 5 ) = 0 . 4, . . . partial knowledge about object types and relations ball ( b 1 ), player ( b 2 ), player ( b 3 ), leg ( b 4 ), leg ( b 5 ), partOf ( b 3 , b 2 ), kicks ( b 2 , b 1 ), hasNum ( b 3 , b 7 ),. . . ontological axioms ∀ xy . partOf ( x , y ) ∧ leg ( x ) → player ( y ), ∀ xy , kick ( x , y ) → player ( x ) ∧ ball ( y ), ∀ xypartOf ( x , y ) → contRatio ( x , y ) > . 9 ∀ xplayer ( x ) → ¬ ball ( x ), Luciano Serafini, Ivan Donadello, Artur d’Avila Garces ( Fondazione Bruno Kessler, Italy University of Trento, Italy City University London, Learning and Reasoning in Logic Tensor Networks May 7, 2017 9 / 23