VoxEL: A Benchmark Dataset for Multilingual Entity Linking Henry - - PowerPoint PPT Presentation

▶

Nov 02, 2023 316 likes •860 views

VoxEL: A Benchmark Dataset for Multilingual Entity Linking Henry Rosales-M endez, Aidan Hogan and Barbara Poblete University of Chile { hrosales,ahogan,bpoblete } @dcc.uchile.cl October 10, 2018 ISWC 2018 - The 17th Internationl

SLIDE 1

VoxEL: A Benchmark Dataset for Multilingual Entity Linking †

Henry Rosales-M´ endez, Aidan Hogan and Barbara Poblete

University of Chile {hrosales,ahogan,bpoblete}@dcc.uchile.cl

October 10, 2018

† ISWC 2018 - The 17th Internationl Semantic Web Conference

SLIDE 2

Example

SLIDE 3

Example - Entity Recognition

SLIDE 4

Example - Entity Disambiguation

SLIDE 5

Applications

Semantic Search
Semantic Annotations
Relation Extraction
Topic Extraction

SLIDE 6

Name Variations in Entity Linking

Michael J. Jackson King of Pop Michael Joseph Jackson

SLIDE 7

Name Variations in Entity Linking

Michael Jackson

SLIDE 8

Multilingual Entity Linking - English

SLIDE 9

Multilingual Entity Linking - Italian

SLIDE 10

Multilingual Entity Linking - Spanish

SLIDE 11

Datasets

SLIDE 12

Datasets

SLIDE 13

Goals

1 Create a benchmark dataset for multilingual Entity Linking

SLIDE 14

Curated source: VoxEurop

SLIDE 15

Example - Any other entity?

SLIDE 16

Example - Any other entity?

SLIDE 17

Example - Any other entity?

SLIDE 18

Example - Any other entity?

SLIDE 19

Example annotations produced by four EL systems

SLIDE 20

Example annotations produced by four EL systems

SLIDE 21

Example annotations produced by four EL systems

Aida

SLIDE 22

Example annotations produced by four EL systems

Aida Babelfy

SLIDE 23

Example annotations produced by four EL systems

Aida Babelfy

DBpedia Spotlight

SLIDE 24

Example annotations produced by four EL systems

Aida Babelfy

DBpedia Spotlight T agME

SLIDE 25

What should Entity Linking link?

SLIDE 26

Datasets

SLIDE 27

Datasets

SLIDE 28

Goals

1 Create a benchmark dataset for multilingual Entity Linking

SLIDE 29

Goals

1 Create a benchmark dataset for multilingual Entity Linking 2 Create two versions of the dataset: strict and relaxed.

SLIDE 30

Strict verison: class-based definition

SLIDE 31

Strict verison: class-based definition

SLIDE 32

Strict verison: class-based definition

?

SLIDE 33

Relaxed version: Knowledge Base definition

SLIDE 34

Creation of VoxEL dataset

It is based on curated text from five languages.
Same sentences by each corresponding document.
Same annotations by each corresponding sentence.
Revision process.

SLIDE 35

Summary

SLIDE 36

Summary

SLIDE 37

Summary

SLIDE 38

Experiments

1 GERBIL Evaluation of state-of-the-art approaches

SLIDE 39

Experiments

SLIDE 40

Experiments

SLIDE 41

Experiments

(a) Results of the Relaxed version of VoxEL DB-sp FREME TAGME

.58 .60 .66 .61

Babefy

r s

Babefy

.58

Avg. of Micro F1

DE EN ES FR IT

.34 .30 .34.33.32 .65 .60 .50.47 .57 .40 .27 .28 .22 .34 .59 .34 .27 .39 .19

THD (b) Results of the Strict version of VoxEL DB-sp FREME TAGME

.76 .78 .81 .74

Babefy

r s

Babefy

.71

Avg. of Micro F1

.64 .70 .72.70.71 .71 .64 .49.51 .65 .81 .54 .60 .53 .74 .86 .75 .60 .72 .50

THD

SLIDE 42

Experiments

(a) Results of the Relaxed version of VoxEL DB-sp FREME TAGME

.58 .60 .66 .61

Babefy

r s

Babefy

.58

Avg. of Micro F1

DE EN ES FR IT

.34 .30 .34.33.32 .65 .60 .50.47 .57 .40 .27 .28 .22 .34 .59 .34 .27 .39 .19

THD (b) Results of the Strict version of VoxEL DB-sp FREME TAGME

.76 .78 .81 .74

Babefy

r s

Babefy

.71

Avg. of Micro F1

.64 .70 .72.70.71 .71 .64 .49.51 .65 .81 .54 .60 .53 .74 .86 .75 .60 .72 .50

THD

SLIDE 43

Experiments

(a) Results of the Relaxed version of VoxEL DB-sp FREME TAGME

.58 .60 .66 .61

Babefy

r s

Babefy

.58

Avg. of Micro F1

DE EN ES FR IT

.34 .30 .34.33.32 .65 .60 .50.47 .57 .40 .27 .28 .22 .34 .59 .34 .27 .39 .19

THD (b) Results of the Strict version of VoxEL DB-sp FREME TAGME

.76 .78 .81 .74

Babefy

r s

Babefy

.71

Avg. of Micro F1

.64 .70 .72.70.71 .71 .64 .49.51 .65 .81 .54 .60 .53 .74 .86 .75 .60 .72 .50

THD

SLIDE 44

Experiments

(a) Results of the Relaxed version of VoxEL DB-sp FREME TAGME

.58 .60 .66 .61

Babefy

r s

Babefy

.58

Avg. of Micro F1

DE EN ES FR IT

.34 .30 .34.33.32 .65 .60 .50.47 .57 .40 .27 .28 .22 .34 .59 .34 .27 .39 .19

THD (b) Results of the Strict version of VoxEL DB-sp FREME TAGME

.76 .78 .81 .74

Babefy

r s

Babefy

.71

Avg. of Micro F1

.64 .70 .72.70.71 .71 .64 .49.51 .65 .81 .54 .60 .53 .74 .86 .75 .60 .72 .50

THD

SLIDE 45

Experiments

1 GERBIL Evaluation of state-of-the-art approaches

SLIDE 46

Experiments

1 GERBIL Evaluation of state-of-the-art approaches 2 Evaluate the performance of state-of-the-art approaches using

machine translation.

SLIDE 47

Experiments

DE EN ES FR IT DE EN ES FR IT Input T ext System Configuration

SLIDE 48

Experiments

DE EN ES FR IT DE EN ES FR IT Input T ext System Configuration

SLIDE 49

Experiments

DE EN ES FR IT DE EN ES FR IT Input T ext System Configuration

SLIDE 50

Experiments

DE EN ES FR IT DE EN ES FR IT Input T ext System Configuration

SLIDE 51

Experiments

(a) Results of the Relaxed version of VoxEL DB-sp FREME TAGME

.51 .52 .55

Babefy

r s

Babefy

Avg. of Micro F1

Calibrated Translation English

.31 .30 .31 .45 .33 .47 .40 .31 .41 .43 .41 .46 .39 .24

THD

.39

(b) Results of the Strict version of VoxEL DB-sp FREME TAGME

.30 .35 .32

Babefy

r s

Babefy

Avg. of Micro F1

.53 .60 .57 .71 .53 .71 .70 .57 .71 .33 .27 .33 .60 .55

THD

.63

SLIDE 52

Conclusion

Our main contribution is VoxEL (https://dx.doi.org/10.6084/m9.figshare.6539675)

Most systems perform (much) better for English.
Machine Translation could be an option to address

multilingual domains in Entity Linking.

SLIDE 53

Poster P20: Machine Translation vs. Multilingual Approaches for Entity Linking

EN IT ES