C r e a t i n g a C o mmo n s e n s e K n o w l - - PowerPoint PPT Presentation

▶
c r e a t i n g a c o mmo n s e n s e k n o w l e d g e b
SMART_READER_LITE
LIVE PREVIEW

C r e a t i n g a C o mmo n s e n s e K n o w l - - PowerPoint PPT Presentation

Jul 03, 2023 177 likes •678 views

C r e a t i n g a C o mmo n s e n s e K n o w l e d g e B a s e a b o u t O b j e c t s V a l e r i o B a s i l e ( U n i v e r s i t y o f T u r i n ) S A I L a b , S

slide-1
SLIDE 1

C r e a t i n g a C

  • mmo

n s e n s e K n

  • w

l e d g e B a s e a b

  • u

t O b j e c t s

V a l e r i

  • B

a s i l e ( U n i v e r s i t y

  • f

T u r i n ) S A I L a b , S i e n a 1 2 / 2 / 2 2

slide-2
SLIDE 2

w h

  • a

mi

V a l e r i

  • B

a s i l e A s s i s t a n t p r

  • f

e s s

  • r

@ U n i t

  • P

r e v i

  • u

s l y :

  • P

h D @ R U G G r

  • n

i n g e n

  • P
  • s

t d

  • c

@ I n r i a

C

  • mp

u t a t i

  • n

a l S e ma n t i c s , S e ma n t i c We b , N a t u r a l L a n g u a g e G e n e r a t i

  • n

, I n f

  • r

ma t i

  • n

E x t r a c t i

  • n

, L i n g u i s t i c A n n

  • t

a t i

  • n

, D i s t r i b u t i

  • n

a l S e ma n t i c s , G e n e r a l K n

  • w

l e d g e B a s e s , G a mi fi c a t i

  • n

, S

  • c

i a l Me d i a , S e n t i me n t A n a l y s i s , L e g a l I n f

  • r

ma t i c s , A r g u me n t Mi n i n g , S

  • c

i a l Me d i a , H a t e S p e e c h , . . .

slide-3
SLIDE 3

Robotics and Artificial Intelligence Objects Linguistics and Semantics Machine Learning and Clustering

T

  • d

a y

slide-4
SLIDE 4
  • I Motivation: The Semantics of Objects
  • II Objects, Knowledge and The Web
  • III Objects, Words and Vectors
  • IV Frames and Prototypical Knowledge
  • V Default Knowledge about Objects

T

  • d

a y

slide-5
SLIDE 5

P a r t I Mo t i v a t i

  • n

: The Semantics of Objects

slide-6
SLIDE 6

5-year CHIST-ERA funded project (2014-2018) 4 EU partners

slide-7
SLIDE 7

Deploy robots in human-inhabited environments. The robots autonomously collect real-world data. We use information available on the Semantic Web to identify the semantics of objects.

slide-8
SLIDE 8
slide-9
SLIDE 9
slide-10
SLIDE 10
slide-11
SLIDE 11

P e r c e p t i

  • n

a n d I d e n t i fi c a t i

  • n

Robot deployments in office environments The robot visits fixed waypoints on the map, taking full 360° RGB-D scans

slide-12
SLIDE 12
  • Object classification
  • Room detection
  • Frame detection
  • Inference
  • ...
slide-13
SLIDE 13

P a r t I I Objects, Knowledge and The Web

slide-14
SLIDE 14

Classification What is (not) an object? What type is an object? What is a room? ... Relations How are objects related? Where is an object? What can I do with an object? ...

O b j e c t K n

  • w

l e d g e

slide-15
SLIDE 15

http://lod-cloud.net/

L i n k e d O p e n D a t a

slide-16
SLIDE 16

W e b K B s

  • D

B p e d i a h u b f

  • r

L O D

  • K

n

  • w

R

  • b

s ma l l e r , ma n u a l l y c r a f t e d , r

  • b
  • t

i c

  • r

i e n t e d

  • C
  • n

c e p t N e t l a r g e , a u t

  • ma

t i c a l l y b u i l t , n

  • t
  • L

O D

slide-17
SLIDE 17

T h e S U N d a t a b a s e

https://groups.csail.mit.edu/vision/SUN/

slide-18
SLIDE 18

W e b K B s

slide-19
SLIDE 19

K e y w

  • r

d L i n k i n g Me t h

  • d

s

Vector-based Contextual disambiguation

  • Run String Match on the keywords
  • Split the missed keywords into tokens
  • Run String Match on the tokens
  • Compute the semantic similarity of each

token-entity with all the previously recognized entities

  • Select the highest scoring token-entity

e.g., basket_of_banana dbr: → Basket

slide-20
SLIDE 20

Results 2,493 objects in DBpedia 679 locations in DBpedia 2,935 object-location relations

T h e S U N d a t a b a s e

Classification Relations

slide-21
SLIDE 21
slide-22
SLIDE 22

P a r t I I I Objects, Words and Vectors

slide-23
SLIDE 23

Problem Classification is good, but relations are sparse

O b j e c t K n

  • w

l e d g e

slide-24
SLIDE 24

D i s t r i b u t i

  • n

a l R e l a t i

  • n

a l H y p

  • t

h e s i s

i s a ( E 1 , A ) i s a ( E 2 , B ) S ( E 1 , E 1 ) R ( A , B ) ? ∧ ∧ →

slide-25
SLIDE 25

S e ma n t i c R e l a t e d n e s s

Washing_machine Ashtray Bathroom 5 2 Bedroom 1 Living_room 1 6

C

  • c

c u r r e n c e ma t r i x S i n g u l a r v a l u e d e c

  • mp
  • s

i t i

  • n

M=U ΣV

*

U k ΣkV k

*=M k

L

  • w
  • r

a n k a p p r

  • x

i ma t i

  • n

NASARI: A Novel Approach to a Semantically-Aware Representation of Items (Camacho-Collados, Pilehvar and Navigli, 2015)

slide-26
SLIDE 26

S e ma n t i c S i mi l a r i t y

B a t h r

  • m

A s h t r a y Wa s h i n g _ ma c h i n e

α β s i m( B a t h r

  • m,

Wa s h i n g _ ma c h i n e ) = c

  • s

() . 7 1 α ≈ s i m( B a t h r

  • m,

A s h t r a y ) = c

  • s

() . 3 7 β ≈

slide-27
SLIDE 27

P l a c e C l a s s i fi c a t i

  • n

= Cosine similarity on NASARI + aggregation, weighting by distance, ...

slide-28
SLIDE 28

E v l u a t i

  • n

: l

  • c

a t e d A t

Gold standard: SUN database linked to DBpedia

slide-29
SLIDE 29

E v l u a t i

  • n

: u s e d F

  • r

Gold standard: ConceptNet linked to DBpedia

slide-30
SLIDE 30

931 high confidence location relations Only 52 were in the gold standard set E.g.: Trivet Kitchen → Flight_bag Airport_lounge → Soap_dispenser Unisex_public_toilet → + many related datasets: https:/ /project.inria.fr/aloof/data/

R e s u l t s

slide-31
SLIDE 31

Object-action relation (usedFor)

Extracting common sense knowledge via triple ranking using supervised and unsupervised distributional models S Jebbara, V Basile, E Cabrio, P Cimiano, Semantic Web 2018

Improving object detection

Semantic web-mining and deep vision for lifelong object discovery J Young, L Kunze, V Basile, E Cabrio, N Hawes, B Caputo Robotics and Automation, ICRA 2017

Object-location relation (locatedAt)

Populating a knowledge base with object-location relations using distributional semantics V Basile, S Jebbara, E Cabrio, P Cimiano, EKAW 2016

D i s t r i b u t i

  • n

a l R e l a t i

  • n

a l H y p

  • t

h e s i s

slide-32
SLIDE 32

P a r t I V Frames and Prototypical Knowledge

slide-33
SLIDE 33

Problem The distributional relational hypothesis is limited to specific relations

slide-34
SLIDE 34

F r a me S e ma n t i c s

F r a me N e t ( 1 9 9 7 ) , F r a me s t e r ( 2 1 6 ) , F r a me b a s e ( 2 1 5 ) Frame name Frame type Frame element role

slide-35
SLIDE 35

F r a me I n s t a n c e

I n s t a n c e i d : < fi 1 2 3 4 5 > F r a me t y p e : f b f r a me : C

  • k

i n g F r a me e l e me n t s :

  • f

b f e : I n s t r u me n t , d b r : K n i f e

  • f

b f e : A g e n t , d b r : P e r s

  • n
  • …

Default Knowledge Prototypical Frame Instances → = F.I. extraction + F.I. clustering

slide-36
SLIDE 36

K n

  • w

l e d g e E x t r a c t i

  • n

h t t p s : / / g i t h u b . c

  • m/

v a l e r i

  • b

a s i l e / l e a r n i n g b y r e a d i n g

Text (Natural Language) Semantic Parsing Word Sense Disambiguation Entity Linking Discourse Representation Structure DBPedia Entities WordNet Synsets Semantic Roles FrameNet Frames Alignment RDF Triples

slide-37
SLIDE 37

F r a me I n s t a n c e E x t r a c t i

  • n
slide-38
SLIDE 38

F r a me S i mi l a r i t y

I n s t a n c e i d : < fi 1 2 3 4 5 > F r a me t y p e : f b f r a me : C

  • k

i n g F r a me e l e me n t s :

  • f

b f e : I n s t r u me n t , d b r : K n i f e

  • f

b f e : A g e n t , d b r : P e r s

  • n
  • …

I n s t a n c e i d : < fi 6 7 8 9 > F r a me t y p e : f b f r a me : E a t i n g F r a me e l e me n t s :

  • f

b f e : I n s t r u me n t , d b r : F

  • r

k

  • f

b f e : A g e n t , d b r : P e r s

  • n
  • …

f r a me t y p e s

slide-39
SLIDE 39

F r a me S i mi l a r i t y

I n s t a n c e i d : < fi 1 2 3 4 5 > F r a me t y p e : f b f r a me : C

  • k

i n g F r a me e l e me n t s :

  • f

b f e : I n s t r u me n t , d b r : K n i f e

  • f

b f e : A g e n t , d b r : P e r s

  • n
  • …

I n s t a n c e i d : < fi 6 7 8 9 > F r a me t y p e : f b f r a me : E a t i n g F r a me e l e me n t s :

  • f

b f e : I n s t r u me n t , d b r : F

  • r

k

  • f

b f e : A g e n t , d b r : P e r s

  • n
  • …

f r a me e l e me n t s

slide-40
SLIDE 40

F r a me S i mi l a r i t y

Me a s u r i n g F r a me I n s t a n c e R e l a t e d n e s s V . B a s i l e , R . L

  • p

e z C

  • n

d

  • r

i , E . C a b r i

  • *

S E M 2 1 8 ( a c c e p t e d )

slide-41
SLIDE 41

P i l

  • t

S t u d y

T e x t f

  • r

l a n g u a g e l e a r n e r s ( 1 , 6 5 3 s h

  • r

t s t

  • r

i e s ) 1 1 4 , 5 3 6 f r a me i n s t a n c e s , 1 5 4 , 4 2 2 f r a me e l e me n t s , 6 8 6 f r a me t y p e s , 2 2 2 r

  • l

e s fi l l e d b y 3 , 3 9 8 t y p e s

  • f

c

  • n

c e p t s . H i e r a r c h i c a l c l u s t e r i n g w i t h

  • u

r d i s t a n c e me t r i c : c

  • mp

l e t e

  • l

i n k a g e a g g l

  • me

r a t i v e ( S c i P y )

F r a me I n s t a n c e E x t r a c t i

  • n

a n d C l u s t e r i n g f

  • r

D e f a u l t K n

  • w

l e d g e B u i l d i n g A . S h a h , V . B a s i l e , E . C a b r i

  • ,

S . K a ma t h S . A p p l i c a t i

  • n

s

  • f

S e ma n t i c We b t e c h n

  • l
  • g

i e s i n R

  • b
  • t

i c s

  • A

N S WE R 1 7

slide-42
SLIDE 42

P i l

  • t

S t u d y

slide-43
SLIDE 43

P i l

  • t

S t u d y

Mo s t f r e q u e n t f r a me t y p e , r

  • l

e a n d e l e me n t f r

  • m

e a c h c l u s t e r

<http://framebase.org/fbframe/Ride_vehicle> <http://framebase.org/fbfe/Vehicle> <http://wordnet−rdf.princeton.edu/wn31/02837983−n>

3 t r i p l e s , a v a i l a b l e a t ~ h t t p : / / p r

  • j

e c t . i n r i a . f r / a l

  • f

/ d a t a / Bicycle

slide-44
SLIDE 44

P a r t V D e f a u l t K n

  • w

l e d g e a b

  • u

t O b j e c t s

slide-45
SLIDE 45

D e f a u l t K n

  • w

l e d g e a b

  • u

t O b j e c t s

RDF dataset of common sense knowledge about

  • bjects.

Object classification, prototypical location, actions, frames... Knowledge extracted from parsing, crowdsourcing, distributional semantics, keyword linking

http:/ /deko.inria.fr/

slide-46
SLIDE 46

D e f a u l t K n

  • w

l e d g e a b

  • u

t O b j e c t s

10,990 nquads (named graphs) 603 from crowdsourcing 1,221 from distributional relational hypothesis 8,046 from keyword kinking 1,120 from KNEWS/frame instance clustering + DeKO ontology

http:/ /deko.inria.fr/

slide-47
SLIDE 47
slide-48
SLIDE 48

L e f t

  • u

t

… b u t

  • p

e n f

  • r

d i s c u s s i

  • n
  • F

u n d a t i

  • n

a l d i s t i n c t i

  • n

s C l a s s v s . I n s t a n c e i n D B p e d i a

A s p r i n

  • L

. , B a s i l e V . , C i a n c a r i n i P . , P r e s u t t i E mp i r i c a l A n a l y s i s

  • f

F

  • u

n d a t i

  • n

a l D i s t i n c t i

  • n

s i n L i n k e d O p e n D a t a ( I J C A I 2 1 8 )

  • N

a t u r e

  • f

p r

  • t
  • t

y p i c a l r e l a t i

  • n

s D e f a u l t L

  • g

i c a n d R D F

slide-49
SLIDE 49

T h e E n d ( Q / A )

slide-50
SLIDE 50
  • Valerio Basile, Elena Cabrio, Fabien Gandon, Debora Nozza:

Mapping Natural Language Labels to Structured Web Resources NL4AI 2018

  • Valerio Basile, Roque Lopez Condori, Elena Cabrio:

Measuring Frame Instance Relatedness. *SEM 2018

  • Soufian Jebbara, Valerio Basile, Elena Cabrio, Philip Cimiano (2018):

Extracting common sense knowledge via triple ranking using supervised and unsupervised distribu tional models Semantic Web

  • Avijit Shah, Valerio Basile, Elena Cabrio, Sowmya Kamath S.:

Frame Instance Extraction and Clustering for Default Knowledge Building. AnSWeR 2017.

  • Jay Young, Valerio Basile, Markus Suchi, Lars Kunze, Nick Hawes, Markus Vincze, Barbara

Caputo: Making sense of indoor spaces using semantic web mining and situated robot perception ESWC 2017

  • Jay Young, Lars Kunze, Valerio Basile, Elena Cabrio, Nick Hawes, Barbara Caputo:

Semantic Web-Mining and Deep Vision for Lifelong Object Discovery. ICRA 2017

  • Valerio Basile, Soufian Jebbara, Elena Cabrio, Philipp Cimiano:

Populating a Knowledge Base with Object-Location Relations Using Distributional Semantics EKAW 2016

  • Jay Young, Valerio Basile, Lars Kunze, Elena Cabrio, Nick Hawes:

Towards Lifelong Object Learning by Integrating Situated Robot Perception and Semantic Web Min ing ECAI 2016

  • Valerio Basile, Elena Cabrio, Fabien Gandon:

Building a General Knowledge Base of Physical Objects for Robots. ESWC 2016