Fine-Grained Evaluation for Entity Linking Henry Rosales-M endez, - - PowerPoint PPT Presentation

▶
fine grained evaluation for entity linking
SMART_READER_LITE
LIVE PREVIEW

Fine-Grained Evaluation for Entity Linking Henry Rosales-M endez, - - PowerPoint PPT Presentation

Feb 04, 2024 196 likes •724 views

Fine-Grained Evaluation for Entity Linking Henry Rosales-M endez, Aidan Hogan and Barbara Poblete University of Chile { hrosales,ahogan,bpoblete } @dcc.uchile.cl Nov 5, 2019 EMNLP-IJCNLP 2019, Hong Kong Example Example - Entity

slide-1
SLIDE 1

Fine-Grained Evaluation for Entity Linking

Henry Rosales-M´ endez, Aidan Hogan and Barbara Poblete

University of Chile {hrosales,ahogan,bpoblete}@dcc.uchile.cl

Nov 5, 2019 † EMNLP-IJCNLP 2019, Hong Kong

slide-2
SLIDE 2

Example

slide-3
SLIDE 3

Example - Entity Recognition

slide-4
SLIDE 4

Example - Entity Disambiguation

slide-5
SLIDE 5

Knowledge Bases

slide-6
SLIDE 6

Name Variations in Entity Linking

Michael J. Jackson King of Pop Michael Joseph Jackson

slide-7
SLIDE 7

Name Variations in Entity Linking

Michael Jackson

slide-8
SLIDE 8

Text collections with noise

slide-9
SLIDE 9

Text collections with noise

slide-10
SLIDE 10

Text collections with noise

slide-11
SLIDE 11

Multilingual Entity Linking - English

slide-12
SLIDE 12

Multilingual Entity Linking - Italian

slide-13
SLIDE 13

Multilingual Entity Linking - Spanish

slide-14
SLIDE 14
  • What should Entity Linking link?
slide-15
SLIDE 15

Example annotations produced by four EL systems

slide-16
SLIDE 16

Example annotations produced by four EL systems

slide-17
SLIDE 17

Example annotations produced by four EL systems

Aida

slide-18
SLIDE 18

Example annotations produced by four EL systems

Aida Babelfy

slide-19
SLIDE 19

Example annotations produced by four EL systems

Aida Babelfy

DBpedia Spotlight

slide-20
SLIDE 20

Example annotations produced by four EL systems

Aida Babelfy

DBpedia Spotlight T agME

slide-21
SLIDE 21

Questionnaire

Gong-Qing Wu, Ying He, and Xuegang Hu. 2018. Entity linking: An issue to extract corresponding entity with knowledge base. IEEE Access, 6:62206231.

slide-22
SLIDE 22

Questionnaire

In an interview with Martin Bashir for the 2003 documentary Living with Michael Jackson, the King of Pop recalled that Joe often sat with a white belt at hand as he and his four siblings rehearsed. Russian daily Kommersant reports that Moscow will supply the Greeks with gas at rock bottom prices as Tsipras prepares to meet the Russian President.

slide-23
SLIDE 23

Questionnaire

slide-24
SLIDE 24

Questionnaire

1 We sent the questionnaire to 321 researchers, of which 232

requests were delivered successfully.

2 We received a total of 36 responses.

slide-25
SLIDE 25

Questionnaire

slide-26
SLIDE 26

Questionnaire

slide-27
SLIDE 27

Goal

1 Allowing a fine-grained evaluation for Entity Linking

slide-28
SLIDE 28

Categories

BASE FORM

Proper Noun Full Name Short Name Extended Name Alias Numeric/Temporal Common Form Pro-Form

PART OF SPEECH

Noun Phrase Singular Plural Adjective Verb Adverb

OVERLAP

None Maximal Intermediate Minimal

REFERENCE

Direct Anaphoric Metaphoric Metonymic Related Descriptive

slide-29
SLIDE 29

Categories

BASE FORM

Proper Noun Full Name Short Name Extended Name Alias Numeric/Temporal Common Form Pro-Form

PART OF SPEECH

Noun Phrase Singular Plural Adjective Verb Adverb

OVERLAP

None Maximal Intermediate Minimal

REFERENCE

Direct Anaphoric Metaphoric Metonymic Related Descriptive

slide-30
SLIDE 30

Categories

BASE FORM

Proper Noun Full Name Short Name Extended Name Alias Numeric/Temporal Common Form Pro-Form

PART OF SPEECH

Noun Phrase Singular Plural Adjective Verb Adverb

OVERLAP

None Maximal Intermediate Minimal

REFERENCE

Direct Anaphoric Metaphoric Metonymic Related Descriptive

slide-31
SLIDE 31

Categories

BASE FORM

Proper Noun Full Name Short Name Extended Name Alias Numeric/Temporal Common Form Pro-Form

PART OF SPEECH

Noun Phrase Singular Plural Adjective Verb Adverb

OVERLAP

None Maximal Intermediate Minimal

REFERENCE

Direct Anaphoric Metaphoric Metonymic Related Descriptive

slide-32
SLIDE 32

Categories

BASE FORM

Proper Noun Full Name Short Name Extended Name Alias Numeric/Temporal Common Form Pro-Form

PART OF SPEECH

Noun Phrase Singular Plural Adjective Verb Adverb

OVERLAP

None Maximal Intermediate Minimal

REFERENCE

Direct Anaphoric Metaphoric Metonymic Related Descriptive

slide-33
SLIDE 33

Categories

BASE FORM

Proper Noun Full Name Short Name Extended Name Alias Numeric/Temporal Common Form Pro-Form

PART OF SPEECH

Noun Phrase Singular Plural Adjective Verb Adverb

OVERLAP

None Maximal Intermediate Minimal

REFERENCE

Direct Anaphoric Metaphoric Metonymic Related Descriptive

slide-34
SLIDE 34

Categories

BASE FORM

Proper Noun Full Name Short Name Extended Name Alias Numeric/Temporal Common Form Pro-Form

PART OF SPEECH

Noun Phrase Singular Plural Adjective Verb Adverb

OVERLAP

None Maximal Intermediate Minimal

REFERENCE

Direct Anaphoric Metaphoric Metonymic Related Descriptive

slide-35
SLIDE 35

Categories

BASE FORM

Proper Noun Full Name Short Name Extended Name Alias Numeric/Temporal Common Form Pro-Form

PART OF SPEECH

Noun Phrase Singular Plural Adjective Verb Adverb

OVERLAP

None Maximal Intermediate Minimal

REFERENCE

Direct Anaphoric Metaphoric Metonymic Related Descriptive

slide-36
SLIDE 36

Categories

BASE FORM

Proper Noun Full Name Short Name Extended Name Alias Numeric/Temporal Common Form Pro-Form

PART OF SPEECH

Noun Phrase Singular Plural Adjective Verb Adverb

OVERLAP

None Maximal Intermediate Minimal

REFERENCE

Direct Anaphoric Metaphoric Metonymic Related Descriptive

slide-37
SLIDE 37

Categories

BASE FORM

Proper Noun Full Name Short Name Extended Name Alias Numeric/Temporal Common Form Pro-Form

PART OF SPEECH

Noun Phrase Singular Plural Adjective Verb Adverb

OVERLAP

None Maximal Intermediate Minimal

REFERENCE

Direct Anaphoric Metaphoric Metonymic Related Descriptive

slide-38
SLIDE 38

Categories

BASE FORM

Proper Noun Full Name Short Name Extended Name Alias Numeric/Temporal Common Form Pro-Form

PART OF SPEECH

Noun Phrase Singular Plural Adjective Verb Adverb

OVERLAP

None Maximal Intermediate Minimal

REFERENCE

Direct Anaphoric Metaphoric Metonymic Related Descriptive

slide-39
SLIDE 39

Categories

BASE FORM

Proper Noun Full Name Short Name Extended Name Alias Numeric/Temporal Common Form Pro-Form

PART OF SPEECH

Noun Phrase Singular Plural Adjective Verb Adverb

OVERLAP

None Maximal Intermediate Minimal

REFERENCE

Direct Anaphoric Metaphoric Metonymic Related Descriptive

slide-40
SLIDE 40

Re-annotation and categorization

1 KORE50: 1 doc, 50 sentences 2 VoxEL: 15 doc, 94 sentences 3 ACE04: first 20 doc, 214 sentences

slide-41
SLIDE 41

Re-annotation and categorization

slide-42
SLIDE 42

Re-annotation and categorization

slide-43
SLIDE 43

Validating by tag

slide-44
SLIDE 44

Validating by tag

slide-45
SLIDE 45

Validating by tag

slide-46
SLIDE 46

How to perform an evaluation for categorized datasets?

slide-47
SLIDE 47

Traditional F1

P = |TP| |S| R = |TP| |G| F1 = 2 · P · R P + R

slide-48
SLIDE 48

Fuzzy set

slide-49
SLIDE 49

Modifications to F1

P = |TP| |S| R∗ =

  • a∈S µG ∗(a)
  • a∈G µG ∗(a)

F1 = 2 · P · R∗ P + R∗

slide-50
SLIDE 50
  • Prop1: the values for R∗ and F ∗

1 both range between 0 and

1, inclusive.

  • Prop2: when µG ∗ : G → {1} (i.e., when memberships are

binary), R∗ and F ∗

1 correspond to R and F1.

  • Prop3: missing annotations with higher membership degree

are penalized more in R∗ and F ∗

1 than those with lower degree.

slide-51
SLIDE 51

Conclusions

  • We stress the lack of consensus about what should Entity

Linking link.

  • We propose a set of categories for Entity Linking.
  • We re-annotate three datasets: VoxEL, KORE50 and

ACE04−first 20.

  • We extend F1 measure.
slide-52
SLIDE 52

Fine-Grained Evaluation for Entity Linking

Henry Rosales-M´ endez, Aidan Hogan and Barbara Poblete

University of Chile {hrosales,ahogan,bpoblete}@dcc.uchile.cl

Nov 5, 2019 † EMNLP-IJCNLP 2019, Hong Kong