C r e a t i n g a C o mmo n s e n s e K n o w l - PowerPoint PPT Presentation

C r e a t i n g a C o mmo n s e n s e K n o w l e d g e B a s e a b o u t O b j e c t s V a l e r i o B a s i l e ( U n i v e r s i t y o f T u r i n ) S A I L a b , S i e n a 1 2 / 2 / 2 0 2 0

w h o a mi V a l e r i o B a s i l e A s s i s t a n t p r o f e s s o r @ U n i t o P r e v i o u s l y : ● P h D @ R U G G r o n i n g e n ● P o s t d o c @ I n r i a C o mp u t a t i o n a l S e ma n t i c s , S e ma n t i c We b , N a t u r a l L a n g u a g e G e n e r a t i o n , I n f o r ma t i o n E x t r a c t i o n , L i n g u i s t i c A n n o t a t i o n , D i s t r i b u t i o n a l S e ma n t i c s , G e n e r a l K n o w l e d g e B a s e s , G a mi fi c a t i o n , S o c i a l Me d i a , S e n t i me n t A n a l y s i s , L e g a l I n f o r ma t i c s , A r g u me n t Mi n i n g , S o c i a l Me d i a , H a t e S p e e c h , . . .

T o d a y Robotics and Artificial Intelligence Objects Linguistics and Semantics Machine Learning and Clustering

T o d a y ● I Motivation: The Semantics of Objects ● II Objects, Knowledge and The Web ● III Objects, Words and Vectors ● IV Frames and Prototypical Knowledge ● V Default Knowledge about Objects

P a r t I Mo t i v a t i o n : The Semantics of Objects

5-year CHIST-ERA funded project (2014-2018) 4 EU partners

Deploy robots in human-inhabited environments. The robots autonomously collect real-world data. We use information available on the Semantic Web to identify the semantics of objects.

P e r c e p t i o n a n d I d e n t i fi c a t i o n Robot deployments in office environments The robot visits fixed waypoints on the map, taking full 360° RGB-D scans

● Object classification ● Room detection ● Frame detection ● Inference ● ...

P a r t I I Objects, Knowledge and The Web

O b j e c t K n o w l e d g e Classification What is (not) an object? What type is an object? What is a room? ... Relations How are objects related? Where is an object? What can I do with an object? ...

L i n k e d O p e n D a t a http://lod-cloud.net/

W e b K B s ● D B p e d i a h u b f o r L O D ● K n o w R o b s ma l l e r , ma n u a l l y c r a f t e d , r o b o t i c - o r i e n t e d ● C o n c e p t N e t l a r g e , a u t o ma t i c a l l y b u i l t , n o t - L O D

T h e S U N d a t a b a s e https://groups.csail.mit.edu/vision/SUN/

W e b K B s

K e y w o r d L i n k i n g Me t h o d s Vector-based Contextual disambiguation ● Run String Match on the keywords ● Split the missed keywords into tokens ● Run String Match on the tokens ● Compute the semantic similarity of each token-entity with all the previously recognized entities ● Select the highest scoring token-entity e.g., basket_of_banana dbr: Basket →

T h e S U N d a t a b a s e Classification Results 2,493 objects in DBpedia 679 locations in DBpedia Relations 2,935 object-location relations

P a r t I I I Objects, Words and Vectors

O b j e c t K n o w l e d g e Problem Classification is good, but relations are sparse

D i s t r i b u t i o n a l R e l a t i o n a l H y p o t h e s i s ∧ ∧ → i s a ( E 1 , A ) i s a ( E 2 , B ) S ( E 1 , E 1 ) R ( A , B ) ?

S e ma n t i c R e l a t e d n e s s C o - o c c u r r e n c e ma t r i x S i n g u l a r v a l u e d e c o mp o s i t i o n Washing_machine Ashtray * M = U ΣV Bathroom 5 2 L o w - r a n k a p p r o x i ma t i o n Bedroom 0 1 * = M k U k Σ k V k Living_room 1 6 NASARI: A Novel Approach to a Semantically-Aware Representation of Items (Camacho-Collados, Pilehvar and Navigli, 2015)

S e ma n t i c S i mi l a r i t y Wa s h i n g _ ma c h i n e B a t h r o o m α β A s h t r a y α ≈ s i m( B a t h r o o m, Wa s h i n g _ ma c h i n e ) = c o s () 0 . 7 1 β ≈ s i m( B a t h r o o m, A s h t r a y ) = c o s () 0 . 3 7

P l a c e C l a s s i fi c a t i o n = Cosine similarity on NASARI + aggregation, weighting by distance, ...

E v l u a t i o n : l o c a t e d A t Gold standard: SUN database linked to DBpedia

E v l u a t i o n : u s e d F o r Gold standard: ConceptNet linked to DBpedia

R e s u l t s 931 high confidence location relations Only 52 were in the gold standard set E.g.: Trivet Kitchen → Flight_bag Airport_lounge → Soap_dispenser Unisex_public_toilet → + many related datasets: https:/ /project.inria.fr/aloof/data/

D i s t r i b u t i o n a l R e l a t i o n a l H y p o t h e s i s Object-action relation (usedFor) Extracting common sense knowledge via triple ranking using supervised and unsupervised distributional models S Jebbara, V Basile, E Cabrio, P Cimiano, Semantic Web 2018 Improving object detection Semantic web-mining and deep vision for lifelong object discovery J Young, L Kunze, V Basile, E Cabrio, N Hawes, B Caputo Robotics and Automation, ICRA 2017 Object-location relation (locatedAt) Populating a knowledge base with object-location relations using distributional semantics V Basile, S Jebbara, E Cabrio, P Cimiano, EKAW 2016

P a r t I V Frames and Prototypical Knowledge

Problem The distributional relational hypothesis is limited to specific relations

F r a me S e ma n t i c s Frame name Frame type role Frame element F r a me N e t ( 1 9 9 7 ) , F r a me s t e r ( 2 0 1 6 ) , F r a me b a s e ( 2 0 1 5 )

F r a me I n s t a n c e I n s t a n c e i d : < fi 1 2 3 4 5 > F r a me t y p e : f b f r a me : C o o k i n g F r a me e l e me n t s : ● f b f e : I n s t r u me n t , d b r : K n i f e ● f b f e : A g e n t , d b r : P e r s o n ● … → Default Knowledge Prototypical Frame Instances = F.I. extraction + F.I. clustering

K n o w l e d g e E x t r a c t i o n Discourse Semantic Semantic Representation Roles Parsing Structure Text RDF Word Sense WordNet FrameNet Alignment (Natural Language) Triples Disambiguation Synsets Frames Entity DBPedia Linking Entities h t t p s : / / g i t h u b . c o m/ v a l e r i o b a s i l e / l e a r n i n g b y r e a d i n g

F r a me I n s t a n c e E x t r a c t i o n

F r a me S i mi l a r i t y f r a me t y p e s I n s t a n c e i d : < fi 1 2 3 4 5 > I n s t a n c e i d : < fi 6 7 8 9 0 > F r a me t y p e : F r a me t y p e : f b f r a me : E a t i n g f b f r a me : C o o k i n g F r a me e l e me n t s : F r a me e l e me n t s : ● f ● f b f e : I n s t r u me n t , b f e : I n s t r u me n t , d b r : F o r k d b r : K n i f e ● f ● f b f e : A g e n t , d b r : P e r s o n b f e : A g e n t , d b r : P e r s o n ● … ● …

F r a me S i mi l a r i t y f r a me e l e me n t s I n s t a n c e i d : < fi 1 2 3 4 5 > I n s t a n c e i d : < fi 6 7 8 9 0 > F r a me t y p e : F r a me t y p e : f b f r a me : E a t i n g f b f r a me : C o o k i n g F r a me e l e me n t s : F r a me e l e me n t s : ● f ● f b f e : I n s t r u me n t , b f e : I n s t r u me n t , d b r : F o r k d b r : K n i f e ● f ● f b f e : A g e n t , d b r : P e r s o n b f e : A g e n t , d b r : P e r s o n ● … ● …

F r a me S i mi l a r i t y Me a s u r i n g F r a me I n s t a n c e R e l a t e d n e s s V . B a s i l e , R . L o p e z C o n d o r i , E . C a b r i o * S E M 2 0 1 8 ( a c c e p t e d )