To Towards Understanding the Ge Geome metry of Knowl wledge Gr - - PowerPoint PPT Presentation

▶
to towards understanding the ge geome metry of knowl
SMART_READER_LITE
LIVE PREVIEW

To Towards Understanding the Ge Geome metry of Knowl wledge Gr - - PowerPoint PPT Presentation

Oct 23, 2023 161 likes •783 views

To Towards Understanding the Ge Geome metry of Knowl wledge Gr Graph Em Embed eddings Chandrahas 1 , Aditya Sharma 2 , Partha Talukdar 1,2 1 Computer Science and Automation 2 Computatational and Data Sciences Indian Institute of Science,

slide-1
SLIDE 1

To Towards Understanding the Ge Geome metry of Knowl wledge Gr Graph Em Embed eddings

Chandrahas1, Aditya Sharma2, Partha Talukdar1,2

1Computer Science and Automation 2Computatational and Data Sciences

Indian Institute of Science, Bangalore

slide-2
SLIDE 2

Kn Knowledge Graphs (KG)

Has Football Team Lionel Messi FC Barcelona Argentina National Football Team

1

slide-3
SLIDE 3

Kn Knowledge Graphs (KG)

Has Football Team Lionel Messi FC Barcelona Argentina National Football Team

Entities

2

slide-4
SLIDE 4

Kn Knowledge Graphs (KG)

Has Football Team Lionel Messi FC Barcelona Argentina National Football Team

Entities Relations

3

slide-5
SLIDE 5

Kn Knowledge Graphs (KG)

4

  • Example KGs
slide-6
SLIDE 6

Kn Knowledge Graphs (KG)

5

  • Example KGs
  • Applications
  • Search
slide-7
SLIDE 7

Kn Knowledge Graphs (KG)

6

  • Example KGs
  • Applications
  • Search
  • Question

Answering

slide-8
SLIDE 8

KG KG Em Embe beddi ddings ngs

  • Represents entities and relations as vectors in a vector space

ℝ𝑒

7

TransE1

  • 1. Translating Embeddings for Modeling Multi-relational Data, Bordes et al.
slide-9
SLIDE 9

KG KG Em Embe beddi ddings ngs

  • Represents entities and relations as vectors in a vector space

ℝ𝑒 𝒆 Plays For ... ... ...

8

TransE1

  • 1. Translating Embeddings for Modeling Multi-relational Data, Bordes et al. NIPS 2013.
slide-10
SLIDE 10

Ge Geometr try of Em Embe beddi ddings ngs

9

  • Arrangement of vectors in the vector space.
slide-11
SLIDE 11

Ge Geometr try of Em Embe beddi ddings ngs

10

  • A recent work by (Mimno and Thompson, 2017)1 presented an

analysis of the geometry of word embeddings and revealed interesting results.

  • However, geometrical understanding of KG embeddings is very

limited, despite their popularity.

  • 1. The strange geometry of skip-gram with negative sampling, Mimno and Thompson, EMNLP 2017
slide-12
SLIDE 12

Pr Problem

  • Study the geometrical behavior of KG embeddings learnt by different

methods.

  • Study the effect of various hyper-parameters used during training on

the geometry of KG embeddings.

  • Study the correlation between the geometry and performance of KG

embeddings.

11

slide-13
SLIDE 13

KG KG Embedding Methods

  • Learns d-dimensional vectors for entities 𝓕 and relations 𝓢 in a KG.

12

slide-14
SLIDE 14

KG KG Embedding Methods

  • Learns d-dimensional vectors for entities 𝓕 and relations 𝓢 in a KG.
  • A score function 𝛕 : 𝓕⨉𝓢⨉𝓕→ℝ distinguishes correct triples 𝑈

+

from incorrect triples 𝑈

−. For example,

𝛕(Messi, plays-for-team, Barcelona) > 𝛕(Messi, plays-for-team, Liverpool)

13

slide-15
SLIDE 15

KG KG Embedding Methods

  • Learns d-dimensional vectors for entities 𝓕 and relations 𝓢 in a KG.
  • A score function 𝛕 : 𝓕⨉𝓢⨉𝓕→ℝ distinguishes correct triples 𝑈

+

from incorrect triples 𝑈

−. For example,

𝛕(Messi, plays-for-team, Barcelona) > 𝛕(Messi, plays-for-team, Liverpool)

  • A loss function 𝑀(𝑈

+, 𝑈 −) is used for training the embeddings (usually

logistic loss or margin-based ranking loss).

14

slide-16
SLIDE 16

KG KG Embedding Methods

15

slide-17
SLIDE 17

KG KG Embedding Methods

  • Additive Methods
  • Multiplicative Methods
  • Neural Methods

16

slide-18
SLIDE 18

KG KG Embedding Methods

17

☉ Entry-wise product ★ Circular correlation

slide-19
SLIDE 19

Ge Geometr tric ical al Metr trics ics

  • Average Vector Length

18

slide-20
SLIDE 20

Ge Geometr tric ical al Metr trics ics

  • Average Vector Length
  • Alignment to Mean

19

slide-21
SLIDE 21

Ge Geometr tric ical al Metr trics ics

  • Conicity

20

slide-22
SLIDE 22

Ge Geometr tric ical al Metr trics ics

  • Conicity
  • Vector Spread

21

slide-23
SLIDE 23

Ge Geometr try of Em Embe beddi ddings ngs

22

High Conicity Low Conicity

slide-24
SLIDE 24

Expe Experiments

  • We study the effect of following factors on the geometry of KG

Embeddings

  • Type of method (Additive or Multiplicative)
  • Number of Negative Samples
  • Dimension of Vector Space
  • We also study the correlation of performance and geometry.
  • For experiments, we used FB15k dataset.

23

slide-25
SLIDE 25

Addi Additive vs s Mul ultipl plicative (En (Entity y Vectors) s)

24

Additive Multiplicative

slide-26
SLIDE 26

Addi Additive vs s Mul ultipl plicative (R (Relation n Vectors) s)

25

Additive Multiplicative

slide-27
SLIDE 27

26

Model Type Conicity Vector Spread Additive Low High Multiplicative High Low

Addi Additive vs s Mul ultipl plicative

slide-28
SLIDE 28

Ef Effect of #Negative Samples (Entity Vectors)

27

slide-29
SLIDE 29

Ef Effect of #Negative Samples (Entity Vectors)

28

Additive

slide-30
SLIDE 30

Ef Effect of #Negative Samples (Entity Vectors)

29

Multiplicative

slide-31
SLIDE 31

Ef Effect of #Negative Samples (Entity Vectors)

30

Additive No change

slide-32
SLIDE 32

Ef Effect of #Negative Samples (Entity Vectors)

31

Additive No change Multiplicative Conicity Increases

slide-33
SLIDE 33

Ef Effect of #Negative Samples (Entity Vectors)

32

Additive No change

slide-34
SLIDE 34

Ef Effect of #Negative Samples (Entity Vectors)

33

Additive No change Multiplicative AVL decreases

slide-35
SLIDE 35

Ef Effect of #Negative Samples

34

Model Type Vector Type Conicity AVL Additive Entity No Change No Change Relation No Change No Change Multiplicative Entity Increases Decreases Relation Decreases No Change except HolE

slide-36
SLIDE 36

SG SGNS S (Word2V 2Vec1) ) as s Multiplicative Model

  • Similar observation was made by (Mimno and Thompson, 2017)2 for

SGNS based word embeddings where higher #negatives resulted in higher conicity.

  • Word2Vec1 maximizes word and context vector dot product for

positive word-context pairs.

  • This behavior is consistent with that of multiplicative models.

35

  • 1. Distributed representations of words and phrases and their compositionality, Mikolov et al. NIPS 2013
  • 2. The strange geometry of skip-gram with negative sampling, Mimno and Thompson, EMNLP 2017
slide-37
SLIDE 37

Ef Effect of #Dimensions (Entity Vectors)

36

Additive No change

slide-38
SLIDE 38

Ef Effect of #Dimensions (Entity Vectors)

37

Additive No change Multiplicative Conicity decreases

slide-39
SLIDE 39

Ef Effect of #Dimensions (Entity Vectors)

38

Additive No change

slide-40
SLIDE 40

Ef Effect of #Dimensions (Entity Vectors)

39

Additive No change Multiplicative AVL Increases

slide-41
SLIDE 41

Ef Effect of #Dimensions

40

Model Type Vector Type Conicity AVL Additive Entity No Change No Change Relation No Change No Change Multiplicative Entity Decreases Increases Relation Decreases Increases

slide-42
SLIDE 42

Co Corr rrelation b/w Geome metry y and Perf rforma rmance

41

slide-43
SLIDE 43

Co Corr rrelation b/w Geome metry y and Perf rforma rmance

42

Additive

slide-44
SLIDE 44

Co Corr rrelation b/w Geome metry y and Perf rforma rmance

43

HolE performs bad with higher negatives

slide-45
SLIDE 45

Co Corr rrelation b/w Geome metry y and Perf rforma rmance

44

Negative Slope- Negative Correlation

slide-46
SLIDE 46

Co Corr rrelation b/w Geome metry y and Perf rforma rmance

45

Negative Slope- Negative Correlation Higher Negatives- Higher Slope Magnitude

slide-47
SLIDE 47

Co Corr rrelation b/w Geome metry y and Perf rforma rmance

46

slide-48
SLIDE 48

Co Corr rrelation b/w Geome metry y and Perf rforma rmance

47

Additive and HolE

slide-49
SLIDE 49

Co Corr rrelation b/w Geome metry y and Perf rforma rmance

48

Positive Slope- Positive Correlation

slide-50
SLIDE 50

Co Corr rrelation b/w Geome metry y and Perf rforma rmance

49

Positive Slope- Positive Correlation Higher Negatives- Higher Slope Magnitude

slide-51
SLIDE 51

Co Corr rrelation b/w Geome metry y and Perf rforma rmance

50

  • Additive:

No correlation between geometry and performance.

  • Multiplicative:

For fixed number of negative samples,

  • Conicity has negative correlation with performance
  • AVL has positive correlation with performance
slide-52
SLIDE 52

Co Conclusi sion and Future Work rks

  • We initiated the study of geometrical behavior of KG embeddings and

presented various insights.

  • Explore whether other entity/relation features (eg entity category)

have any correlation with geometry.

  • Explore other geometrical metrics which have better correlation with

performance and use it for learning better KG embeddings.

51

slide-53
SLIDE 53

Ac Ackno knowl wledg dgements

  • We thank Google for the travel grant for attending ACL 2018.
  • We thank MHRD India, Intel, Intuit, Google and Accenture for

supporting our work.

  • We thank the reviewers for their constructive comments.

52

slide-54
SLIDE 54

Th Thank you

53

slide-55
SLIDE 55

Ef Effect of #Negative Samples (Relation Vectors)

54

Additive No change

slide-56
SLIDE 56

Ef Effect of #Negative Samples (Relation Vectors)

55

Additive No change Multiplicative Conicity decreases

slide-57
SLIDE 57

Ef Effect of #Negative Samples (Relation Vectors)

56

Additive No change

slide-58
SLIDE 58

Ef Effect of #Negative Samples (Relation Vectors)

57

Additive No change Multiplicative No change except HolE

slide-59
SLIDE 59

Ef Effect of #Dimensions (Relation Vectors)

58

Additive No change

slide-60
SLIDE 60

Ef Effect of #Dimensions (Relation Vectors)

59

Additive No change Multiplicative Conicity decreases

slide-61
SLIDE 61

Ef Effect of #Dimensions (Relation Vectors)

60

Additive No change

slide-62
SLIDE 62

Ef Effect of #Dimensions (Relation Vectors)

61

Additive No change Multiplicative AVL Increases