KBP 2017 Cold Start KB Construction and Slot Filling Hoa Dang - - PowerPoint PPT Presentation

▶
kbp 2017 cold start
SMART_READER_LITE
LIVE PREVIEW

KBP 2017 Cold Start KB Construction and Slot Filling Hoa Dang - - PowerPoint PPT Presentation

Dec 19, 2022 320 likes •811 views

KBP 2017 Cold Start KB Construction and Slot Filling Hoa Dang Shahzad Rajput U.S. National Institute of Standards and Technology TAC 2017 Hoa Dang, Shahzad Rajput (NIST) KBP 2017 Cold Start TAC 2017 1 / 48 Outline Introduction 1 Task

slide-1
SLIDE 1

KBP 2017 Cold Start

KB Construction and Slot Filling Hoa Dang Shahzad Rajput

U.S. National Institute of Standards and Technology

TAC 2017

Hoa Dang, Shahzad Rajput (NIST) KBP 2017 Cold Start TAC 2017 1 / 48

slide-2
SLIDE 2

Outline

1

Introduction Task Variants Changes in 2017

2

SF/KB Evaluation Definitions Queries Participants Results

3

Conclusion

Hoa Dang, Shahzad Rajput (NIST) KBP 2017 Cold Start TAC 2017 2 / 48

slide-3
SLIDE 3

Outline

1

Introduction Task Variants Changes in 2017

2

SF/KB Evaluation Definitions Queries Participants Results

3

Conclusion

Hoa Dang, Shahzad Rajput (NIST) KBP 2017 Cold Start TAC 2017 3 / 48

slide-4
SLIDE 4

Task Variants

Knowledge Base Construction (K=Many) - Queries were not provided

ED Evaluation Event Argument Evaluation Event Nugget Evaluation BeST Evaluation SF Evaluation (K=1,3)

Slot Filling (K=1) - Queries involving SF slots were provided

Hoa Dang, Shahzad Rajput (NIST) KBP 2017 Cold Start TAC 2017 4 / 48

slide-5
SLIDE 5

Task Variants

Queries - Example # 1

LDC Query

<query id=‘‘CS17_9999 ’’> <mentions > <mention > <name >June McCarthy </name > <docid >ENG_142 </docid > <beg >16931 </beg > <end >16943 </end > </mention > <mention > <name >Junio McCarthy </name > <docid >SPA_142 </docid > <beg >2863 </beg > <end >2869 </end > </mention > </mentions > <enttype >per </ enttype > <nodeid >per_049 </ nodeid > <slot0 >per:children </slot0 > <slot1 >per:age </slot1 > </query >

SF Query

<query id=‘‘ CSSF17_ENG_abcabdefde ’’> <name >June McCarthy </name > <docid >ENG_142 </docid > <beg >16931 </beg > <end >16943 </end > <enttype >PER </ enttype > <slot >per:children </slot > <slot0 >per:children </slot0 > <slot1 >per:age </slot1 > </query > <query id=‘‘ CSSF17_SPA_defdeabcab ’’> <name >Junio McCarthy <</name > <docid >SPA_142 </docid > <beg >2863 </beg > <end >2869 </end > <enttype >PER </ enttype > <slot >per:children </slot > <slot0 >per:children </slot0 > <slot1 >per:age </slot1 > </query > Hoa Dang, Shahzad Rajput (NIST) KBP 2017 Cold Start TAC 2017 5 / 48

slide-6
SLIDE 6

Task Variants

Queries - Example # 2

LDC Query

<query id=‘‘CS17_9999 ’’> <mentions > <mention > <name >June McCarthy </name > <docid >ENG_142 </docid > <beg >16931 </beg > <end >16943 </end > </mention > <mention > <name >Junio McCarthy </name > <docid >SPA_142 </docid > <beg >2863 </beg > <end >2869 </end > </mention > </mentions > <enttype >per </ enttype > <nodeid >per_049 </ nodeid > <slot0 >per:contact.meet_entity </slot0 > </query >

SF Query

<query id=‘‘ CSSF17_ENG_abcabdefde ’’> <name >June McCarthy </name > <docid >ENG_142 </docid > <beg >16931 </beg > <end >16943 </end > <enttype >PER </ enttype > <slot >per:contact.meet_entity </slot > <slot0 >per:contact.meet_entity </slot0 > </query > <query id=‘‘ CSSF17_SPA_defdeabcab ’’> <name >Junio McCarthy <</name > <docid >SPA_142 </docid > <beg >2863 </beg > <end >2869 </end > <enttype >PER </ enttype > <slot >per:contact.meet_entity </slot > <slot0 >per:contact.meet_entity </slot0 > </query > Hoa Dang, Shahzad Rajput (NIST) KBP 2017 Cold Start TAC 2017 6 / 48

slide-7
SLIDE 7

Task Variants

Queries - Example # 3

LDC Query

<query id=‘‘CS17_9999 ’’> <mentions > <mention > <name >June McCarthy </name > <docid >ENG_142 </docid > <beg >16931 </beg > <end >16943 </end > </mention > <mention > <name >Junio McCarthy </name > <docid >SPA_142 </docid > <beg >2863 </beg > <end >2869 </end > </mention > </mentions > <enttype >per </ enttype > <nodeid >per_049 </ nodeid > <slot0 >per:children </slot0 > <slot1 >per:contact.meet_entity </slot1 > </query >

SF Query

<query id=‘‘ CSSF17_ENG_abcabdefde ’’> <name >June McCarthy </name > <docid >ENG_142 </docid > <beg >16931 </beg > <end >16943 </end > <enttype >PER </ enttype > <slot >per:children </slot > <slot0 >per:children </slot0 > <slot1 >per:contact.meet_entity </slot1 > </query > <query id=‘‘ CSSF17_SPA_defdeabcab ’’> <name >Junio McCarthy <</name > <docid >SPA_142 </docid > <beg >2863 </beg > <end >2869 </end > <enttype >PER </ enttype > <slot >per:children </slot > <slot0 >per:children </slot0 > <slot1 >per:contact.meet_entity </slot1 > </query > Hoa Dang, Shahzad Rajput (NIST) KBP 2017 Cold Start TAC 2017 7 / 48

slide-8
SLIDE 8

Task Variants

Queries - Example # 4

LDC Query

<query id=‘‘CS17_9999 ’’> <mentions > <mention > <name >June McCarthy </name > <docid >ENG_142 </docid > <beg >16931 </beg > <end >16943 </end > </mention > <mention > <name >Junio McCarthy </name > <docid >SPA_142 </docid > <beg >2863 </beg > <end >2869 </end > </mention > </mentions > <enttype >per </ enttype > <nodeid >per_049 </ nodeid > <slot0 >per:is_liked_by </slot0 > <slot1 >per:age </slot1 > </query >

SF Query

<query id=‘‘ CSSF17_ENG_abcabdefde ’’> <name >June McCarthy </name > <docid >ENG_142 </docid > <beg >16931 </beg > <end >16943 </end > <enttype >PER </ enttype > <slot >per:is_liked_by </slot > <slot0 >per:is_liked_by </slot0 > <slot1 >per:age </slot1 > </query > <query id=‘‘ CSSF17_SPA_defdeabcab ’’> <name >Junio McCarthy <</name > <docid >SPA_142 </docid > <beg >2863 </beg > <end >2869 </end > <enttype >PER </ enttype > <slot >per:is_liked_by </slot > <slot0 >per:is_liked_by </slot0 > <slot1 >per:age </slot1 > </query > Hoa Dang, Shahzad Rajput (NIST) KBP 2017 Cold Start TAC 2017 8 / 48

slide-9
SLIDE 9

Task Variants

Example#01: KB → EDL

Knowledge Base :e4 type PER :e4 mention "Bart" Doc726 :37 -40 :e4 nominal_mention "brother" Doc726 :15 -21 EDL component EKB :e4_M1 Bart Doc726 :37 -40 NIL_0001 PER NAM 1.0 EKB :e4_M2 brother Doc726 :15 -21 NIL_0001 PER NOM 1.0 Hoa Dang, Shahzad Rajput (NIST) KBP 2017 Cold Start TAC 2017 9 / 48

slide-10
SLIDE 10

Task Variants

Example#02: KB → EAL

Knowledge Base :en1 type GPE :en1 mention " Springfield " Doc726 :402 -412 1.0 :ev1 type CONFLICT.ATTACK :ev1 mention.actual "bombing" Doc726 :418 -424 1.0 :ev1 conflict.attack:place.actual :en1 Doc726 :261 -431; Doc726 :402 -407; NIL 1.0 ... EAL component (arguments) EKB:ev1 .1 Doc726 Conflict.Attack Place Springfield 402 -407 261 -431 402 -407 NIL Actual 1.0 ... EAL component (linking) EKB:ev1.Doc726 EKB:ev1 .1 ... EAL component (corpuslinking) EKB:ev1 EKB:ev1.Doc726 ... Hoa Dang, Shahzad Rajput (NIST) KBP 2017 Cold Start TAC 2017 10 / 48

slide-11
SLIDE 11

Task Variants

Example#03: KB → EN

Knowledge Base :ev1 type CONFLICT.ATTACK :ev1 mention.actual "bombing" Doc726 :418 -424 1.0 EN component # BeginOfDocument Doc726 Doc726 E1 418 ,424 bombing Conflict_Attack Actual # EndOfDocument Hoa Dang, Shahzad Rajput (NIST) KBP 2017 Cold Start TAC 2017 11 / 48

slide-12
SLIDE 12

Task Variants

Example#04: KB → BeST

Knowledge Base :en1 type GPE :en1 mention " Springfield " Doc726 :402 -412 1.0 :en4 type PER :en4 mention "Bart" Doc726 :37 -40 :en4 per:dislikes :en1 Doc726 :422 -427 1.0 BeST component <?xml version ="1.0" encoding ="UTF -8"? > <belief_sentiment_doc id=" Doc726"> <sentiment_annotations > <entities > <entity ere_id =": en1 .500001"

  • ffset ="402"

length ="11" > <text >Springfield </text > <sentiments > <sentiment polarity =" neg" sarcasm ="no" confidence ="1.0" > <source ere_id =": en4 .500001"

  • ffset ="37"

length ="4" >Bart </ source > </sentiment > </sentiments > </entity > </entities > </ sentiment_annotations > </ belief_sentiment_doc > Hoa Dang, Shahzad Rajput (NIST) KBP 2017 Cold Start TAC 2017 12 / 48

slide-13
SLIDE 13

Task Variants

Example#05: KB → Slot-filling

Knowledge Base :en7 type PER :en7 canonical_mention "Bart" Doc726 :434 -437 1.0 :en10 type PER :en10 mention "Lisa" Doc726 :684 -687 1.0 :en10 per:siblings :en7 Doc726 :561 -577 0.7 SF Query <query id=" CS17_ENG_54d83ec119 "> <name >Lisa </name > <docid >Doc726 </docid > <beg >684 </beg > <end >687 </end > <enttype >per </ enttype > <slot0 >per:siblings </slot0 > </query > Slot Filling CS17_ENG_54d83ec119 per:siblings myrun Doc726 :561 -577 Bart PER Doc726 :434 -437 0.7 Hoa Dang, Shahzad Rajput (NIST) KBP 2017 Cold Start TAC 2017 13 / 48

slide-14
SLIDE 14

Changes in 2017

Events and sentiments merged into the KB Multiple justifications (K=1 and K=3) AP-based scoring

Hoa Dang, Shahzad Rajput (NIST) KBP 2017 Cold Start TAC 2017 14 / 48

slide-15
SLIDE 15

Outline

1

Introduction Task Variants Changes in 2017

2

SF/KB Evaluation Definitions Queries Participants Results

3

Conclusion

Hoa Dang, Shahzad Rajput (NIST) KBP 2017 Cold Start TAC 2017 15 / 48

slide-16
SLIDE 16

SF/KB Definitions

Scoring - F1

Wrong or ineXact is Spurious Hop 1 filler whose Hop 0 parent filler is Wrong or ineXact, is Spurious Correct responses are grouped into equivalence classes (EC). At most

  • ne response is Right; all other Spurious

NAM mention in EC, or NOM mentions and the EC is NOM, then

  • ne is Right; otherwise, if only NOM mentions in a NAM EC, then
  • ne is Ignored

Reference = number of single-valued pseudo-slots with a correct response + number of equivalence classes for all list-valued pseudo-slots Recall = #Right / Reference Precision = #Right / (#Right + #Spurious) F1 = 2 * Precision * Recall / (Precision + Recall) Applied only to queries with a known correct answer

Hoa Dang, Shahzad Rajput (NIST) KBP 2017 Cold Start TAC 2017 16 / 48

slide-17
SLIDE 17

SF/KB Definitions

Scoring - AP

===================================================================================== QUERY_ID: CSSF17_ENG_afddc4ed21 LEVEL: 1 AP: 0.3750 = (0.6667 + 1.6667/2) /4 NUM_GROUND_TRUTH : 4 GROUND TRUTH: CSSF17_ENG_afddc4ed21 :1:1 CSSF17_ENG_afddc4ed21 :1:2 CSSF17_ENG_afddc4ed21 :2:1 CSSF17_ENG_afddc4ed21 :2:2 RANKING: ........

  • R. NODEID

CONF. MAPPED_EC V

  • 1

CSSF17_ENG_afddc4ed21 :Entity102:Entity111 0.7396 CSSF17_ENG_afddc4ed21 :1:2 0.6667 2 CSSF17_ENG_afddc4ed21 :Entity102:Entity110 0.6001 CSSF17_ENG_afddc4ed21 :1:1 1.0000 3 CSSF17_ENG_afddc4ed21 :Entity103:Entity111 0.4653 - 0.0000 4 CSSF17_ENG_afddc4ed21 :Entity104:Entity110 0.4581 - 0.0000 5 CSSF17_ENG_afddc4ed21 :Entity104:Entity111 0.4513 - 0.0000 6 CSSF17_ENG_afddc4ed21 :Entity103:Entity110 0.4172 - 0.0000 ===================================================================================== Hoa Dang, Shahzad Rajput (NIST) KBP 2017 Cold Start TAC 2017 17 / 48

slide-18
SLIDE 18

SF/KB Definitions

Aggregates and Projections P/R/F1 Score Variants Aggregates Reported Micro-average Macro-average SF Yes Yes LDC-MAX Yes Yes LDC-MEAN No Yes AP Score Variants Aggregates Reported Micro-average Macro-average SF No Yes LDC-MAX No No LDC-MEAN No Yes SF: consider all entrypoints as a separate query. LDC-MAX: Considering the run’s best entrypoint per LDC query on the basis of F1 score across both hops. LDC-MEAN: Precision, Recall, and F1 for each LDC query is the mean Precision, mean Recall, and mean F1 for all entrypoints for that LDC query.

Hoa Dang, Shahzad Rajput (NIST) KBP 2017 Cold Start TAC 2017 18 / 48

slide-19
SLIDE 19

SF/KB Definitions

Aggregates

Micro-averages are computed as: Total Precision = Total Right Total Right + Total Wrong Total Recall = Total Right Total GT Total F1 = 2 × Total Precision × Total Recall Total Precision + Total Recall Macro-averages are computed as the mean Precision, mean Recall, and mean F1.

Hoa Dang, Shahzad Rajput (NIST) KBP 2017 Cold Start TAC 2017 19 / 48

slide-20
SLIDE 20

Queries

Slot-types

LDC Queries (Queries restricted to) Slot-types Total Slot-filling Events Sentiments Developed 839 315 238 1392 Pooled 266 187 124 577 SF Queries (Queries restricted to) Slot-types Total Slot-filling Events Sentiments Developed 4063 1528 1150 6741 Pooled 1313 913 602 2828

Hoa Dang, Shahzad Rajput (NIST) KBP 2017 Cold Start TAC 2017 20 / 48

slide-21
SLIDE 21

Queries

Languages

LDC Queries Language English Spanish Chinese Developed 1300 1198 1112 Pooled 551 509 494 SF Queries Language Total English Spanish Chinese Developed 3282 1870 1589 6741 Pooled 1337 780 711 2828

Hoa Dang, Shahzad Rajput (NIST) KBP 2017 Cold Start TAC 2017 21 / 48

slide-22
SLIDE 22

Queries

LDC Queries - Slot-filling Slots

10 20 30 40 50 60 70 80 gpe:births_in_city gpe:births_in_stateorprovince gpe:deaths_in_city gpe:deaths_in_country gpe:deaths_in_stateorprovince gpe:employees_or_members gpe:headquarters_in_city gpe:headquarters_in_country gpe:headquarters_in_stateorprovince gpe:holds_shares_in gpe:member_of gpe:organizations_founded gpe:residents_of_city gpe:residents_of_stateorprovince gpe:subsidiaries

  • rg:alternate_names
  • rg:city_of_headquarters
  • rg:country_of_headquarters
  • rg:date_dissolved
  • rg:date_founded
  • rg:employees_or_members
  • rg:founded_by
  • rg:holds_shares_in
  • rg:member_of
  • rg:members
  • rg:number_of_employees_members
  • rg:organizations_founded
  • rg:parents
  • rg:political_religious_affiliation
  • rg:shareholders
  • rg:stateorprovince_of_headquarters
  • rg:students
  • rg:subsidiaries
  • rg:top_members_employees
  • rg:website

per:age per:alternate_names per:cause_of_death per:charges per:children per:cities_of_residence per:city_of_birth per:city_of_death per:countries_of_residence per:country_of_birth per:country_of_death per:date_of_birth per:date_of_death per:employee_or_member_of per:holds_shares_in per:organizations_founded per:origin per:other_family per:parents per:religion per:schools_attended per:siblings per:spouse per:stateorprovince_of_birth per:stateorprovince_of_death per:statesorprovinces_of_residence per:title per:top_member_employee_of Pooled Developed

Hoa Dang, Shahzad Rajput (NIST) KBP 2017 Cold Start TAC 2017 22 / 48

slide-23
SLIDE 23

Queries

LDC Queries - Event Slots

5 10 15 20 25 30 35 40 45 fac:contact.meet_place fac:justice.arrest-jail_place fac:life.injure_place fac:movement.transport-person_destination gpe:conflict.attack_attacker gpe:conflict.attack_place gpe:contact.correspondence_place gpe:contact.meet_entity gpe:contact.meet_place gpe:justice.arrest-jail_place gpe:life.injure_agent gpe:life.injure_place gpe:manufacture.artifact_agent gpe:manufacture.artifact_place gpe:movement.transport-artifact_destination gpe:movement.transport-person_agent gpe:movement.transport-person_destination gpe:movement.transport-person_origin gpe:transaction.transfer-ownership_place loc:conflict.attack_place loc:contact.correspondence_place loc:life.injure_place loc:movement.transport-artifact_destination loc:movement.transport-artifact_origin loc:movement.transport-person_destination loc:movement.transport-person_origin loc:transaction.transfer-ownership_place

  • rg:conflict.attack_attacker
  • rg:conflict.attack_target
  • rg:contact.correspondence_entity
  • rg:contact.meet_entity
  • rg:justice.arrest-jail_agent
  • rg:life.injure_agent
  • rg:manufacture.artifact_agent
  • rg:movement.transport-artifact_agent
  • rg:movement.transport-person_agent
  • rg:transaction.transfer-money_beneficiary
  • rg:transaction.transfer-money_giver
  • rg:transaction.transfer-money_recipient
  • rg:transaction.transfer-ownership_beneficiary
  • rg:transaction.transfer-ownership_giver
  • rg:transaction.transfer-ownership_recipient
  • rg:transaction.transfer-ownership_thing

per:conflict.attack_attacker per:conflict.attack_target per:contact.correspondence_entity per:contact.meet_entity per:justice.arrest-jail_agent per:justice.arrest-jail_person per:life.injure_agent per:life.injure_victim per:manufacture.artifact_agent per:movement.transport-artifact_agent per:movement.transport-person_agent per:movement.transport-person_person per:transaction.transfer-money_beneficiary per:transaction.transfer-money_giver per:transaction.transfer-money_recipient per:transaction.transfer-ownership_beneficiary per:transaction.transfer-ownership_recipient Pooled Developed

Hoa Dang, Shahzad Rajput (NIST) KBP 2017 Cold Start TAC 2017 23 / 48

slide-24
SLIDE 24

Queries

LDC Queries - Sentiment Slots

5 10 15 20 25 30 35 fac:dislikes fac:is_disliked_by fac:is_liked_by fac:likes gpe:dislikes gpe:is_disliked_by gpe:is_liked_by loc:dislikes loc:is_disliked_by loc:is_liked_by loc:likes

  • rg:dislikes
  • rg:is_disliked_by
  • rg:is_liked_by
  • rg:likes

per:dislikes per:is_disliked_by per:is_liked_by per:likes Pooled Developed

Hoa Dang, Shahzad Rajput (NIST) KBP 2017 Cold Start TAC 2017 24 / 48

slide-25
SLIDE 25

Participants

SF/KB Teams - Submissions with respect to languages

Cold Start 2017

Teams KB runs SF runs Tot. ENG SPA CMN XLING Tot. ENG SPA CMN XLING Tot. A2KD Adept 4

  • 4

2 10

  • 10

ICTCAS OKN 5

  • 5
  • 5

ISCAS Sogou

  • 5
  • 5
  • 5

LDC 1 1 1 1 4

  • 4

SAFT ISI 1 1 1

  • 3
  • 3

STANFORD 5 2 4 1 12 4 2 4 1 11 23 TinkerBell 5 5 5 5 20

  • 20

UNIST SAIL

  • 2
  • 2

2 Y dcd zju

  • 3
  • 3

3 dws uma

  • 1
  • 1

1 hltcoe 4

  • 3

4 11

  • 11

newbie mr 01 1

  • 1
  • 1

Total 26 9 23 13 71 10 2 4 1 17 88

Cold Start 2016

Teams KB runs SF runs Tot. ENG SPA CMN XLING Tot. ENG SPA CMN XLING Tot. Total 28 9 6 13 56 43 7 7 10 67 123 Hoa Dang, Shahzad Rajput (NIST) KBP 2017 Cold Start TAC 2017 25 / 48

slide-26
SLIDE 26

Participants

SF/KB Teams - Submissions with respect to slot types

How many submissions had at least one response for a hop-0 query with the given slot type?

Teams KB runs SF runs Slot-filling Events Sentiments Slot-filling Events Sentiments A2KD Adept 10 10

  • ICTCAS OKN

5

  • 2
  • ISCAS Sogou

5 5 5

  • LDC

4 4 4

  • SAFT ISI

1 3 3

  • STANFORD

12

  • 11
  • TinkerBell

20 20 20

  • UNIST SAIL
  • 2
  • Y dcd zju
  • 3
  • dws uma
  • 1
  • hltcoe

11

  • newbie mr 01

1

  • Hoa Dang, Shahzad Rajput (NIST)

KBP 2017 Cold Start TAC 2017 26 / 48

slide-27
SLIDE 27

SF/KB Results

K1 - LDC-MEAN-ALL-Macro - Hop 0 - F1 (SF-SLOTS-only) - English

0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1

ENG-LDC-MEAN-ALL-Macro-Hop0-SF-SLOTS-only-K1

LDC/KB/1 STANFORD/KB/3 TinkerBell/KB/4 hltcoe/KB/4 UNIST_SAIL/SF/2 A2KD_Adept/KB/3 Y_dcd_zju/SF/1 ICTCAS_OKN/KB/3 SAFT_ISI/KB/1 dws_uma/SF/1 newbie_mr_01/KB/1

Precision Recall F1 Hoa Dang, Shahzad Rajput (NIST) KBP 2017 Cold Start TAC 2017 27 / 48

slide-28
SLIDE 28

SF/KB Results

K3 - LDC-MEAN-ALL-Macro - Hop 0 - AP (SF-SLOTS-only) - English

0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1

ENG-LDC-MEAN-ALL-Macro-Hop0-AP-SF-SLOTS-only-K3

LDC/KB/1 STANFORD/KB/3 TinkerBell/KB/3 hltcoe/KB/4 A2KD_Adept/KB/4 SAFT_ISI/KB/1 ICTCAS_OKN/KB/3 newbie_mr_01/KB/1

AP Hoa Dang, Shahzad Rajput (NIST) KBP 2017 Cold Start TAC 2017 28 / 48

slide-29
SLIDE 29

SF/KB Results

K1 - LDC-MEAN-ALL-Macro - Hop 1 - F1 (SENTIMENT-SLOTS-only) - Chinese

0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1

CMN-LDC-MEAN-ALL-Macro-Hop1-SENTIMENT-SLOTS-only-K1

TinkerBell/KB/4 LDC/KB/1 SAFT_ISI/KB/1 ISCAS_Sogou/KB/4

Precision Recall F1 Hoa Dang, Shahzad Rajput (NIST) KBP 2017 Cold Start TAC 2017 29 / 48

slide-30
SLIDE 30

SF/KB Results

K1 - LDC-MEAN-ALL-Macro - Hop 1 - F1 (SF-SLOTS-only) - Cross-lingual

0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1

XLING-LDC-MEAN-ALL-Macro-Hop1-SF-SLOTS-only-K1

TinkerBell/KB/4 LDC/KB/1 STANFORD/KB/1 hltcoe/KB/3 A2KD_Adept/KB/2

Precision Recall F1 Hoa Dang, Shahzad Rajput (NIST) KBP 2017 Cold Start TAC 2017 30 / 48

slide-31
SLIDE 31

SF/KB Results

Layout

0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 ENG-LDC-MAX-ALL-Macro-Hop0-ALL-SLOT-TYPES-K1 LDC/KB/1 A2KD_Adept/KB/2 STANFORD/KB/3 TinkerBell/KB/4 hltcoe/KB/4 SAFT_ISI/KB/1 ICTCAS_OKN/KB/3 newbie_mr_01/KB/1 Precision Recall F1 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 ENG-LDC-MAX-ALL-Macro-Hop1-ALL-SLOT-TYPES-K1 LDC/KB/1 TinkerBell/KB/4 STANFORD/KB/3 hltcoe/KB/2 A2KD_Adept/KB/3 SAFT_ISI/KB/1 ICTCAS_OKN/KB/3 Precision Recall F1 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 ENG-LDC-MAX-ALL-Macro-HopALL-ALL-SLOT-TYPES-K1 LDC/KB/1 TinkerBell/KB/4 STANFORD/KB/3 A2KD_Adept/KB/1 hltcoe/KB/4 SAFT_ISI/KB/1 ICTCAS_OKN/KB/3 newbie_mr_01/KB/1 Precision Recall F1 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 SPA-LDC-MAX-ALL-Macro-Hop0-ALL-SLOT-TYPES-K1 LDC/KB/1 TinkerBell/KB/3 STANFORD/KB/2 SAFT_ISI/KB/1 Precision Recall F1 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 SPA-LDC-MAX-ALL-Macro-Hop1-ALL-SLOT-TYPES-K1 LDC/KB/1 STANFORD/KB/2 TinkerBell/KB/3 Precision Recall F1 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 SPA-LDC-MAX-ALL-Macro-HopALL-ALL-SLOT-TYPES-K1 LDC/KB/1 TinkerBell/KB/3 STANFORD/KB/2 SAFT_ISI/KB/1 Precision Recall F1 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 CMN-LDC-MAX-ALL-Macro-Hop0-ALL-SLOT-TYPES-K1 LDC/KB/1 TinkerBell/KB/2 A2KD_Adept/KB/4 ISCAS_Sogou/KB/4 STANFORD/KB/1 hltcoe/KB/2 SAFT_ISI/KB/1 Precision Recall F1 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 CMN-LDC-MAX-ALL-Macro-Hop1-ALL-SLOT-TYPES-K1 LDC/KB/1 TinkerBell/KB/4 ISCAS_Sogou/KB/1 STANFORD/KB/1 SAFT_ISI/KB/1 A2KD_Adept/KB/4 hltcoe/KB/2 Precision Recall F1 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 CMN-LDC-MAX-ALL-Macro-HopALL-ALL-SLOT-TYPES-K1 LDC/KB/1 TinkerBell/KB/2 ISCAS_Sogou/KB/4 A2KD_Adept/KB/4 STANFORD/KB/1 hltcoe/KB/2 SAFT_ISI/KB/1 Precision Recall F1 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 XLING-LDC-MAX-ALL-Macro-Hop0-ALL-SLOT-TYPES-K1 LDC/KB/1 TinkerBell/KB/5 STANFORD/KB/1 A2KD_Adept/KB/2 hltcoe/KB/2 Precision Recall F1 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 XLING-LDC-MAX-ALL-Macro-Hop1-ALL-SLOT-TYPES-K1 LDC/KB/1 TinkerBell/KB/4 STANFORD/KB/1 hltcoe/KB/3 A2KD_Adept/KB/2 Precision Recall F1 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 XLING-LDC-MAX-ALL-Macro-HopALL-ALL-SLOT-TYPES-K1 LDC/KB/1 TinkerBell/KB/4 STANFORD/KB/1 hltcoe/KB/2 A2KD_Adept/KB/2 Precision Recall F1

Hoa Dang, Shahzad Rajput (NIST) KBP 2017 Cold Start TAC 2017 31 / 48

ENGLISH SPANISH CHINESE CROSS-LINGUAL

slide-32
SLIDE 32

SF/KB Results

Layout

0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 ENG-LDC-MAX-ALL-Macro-Hop0-ALL-SLOT-TYPES-K1 LDC/KB/1 A2KD_Adept/KB/2 STANFORD/KB/3 TinkerBell/KB/4 hltcoe/KB/4 SAFT_ISI/KB/1 ICTCAS_OKN/KB/3 newbie_mr_01/KB/1 Precision Recall F1 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 ENG-LDC-MAX-ALL-Macro-Hop1-ALL-SLOT-TYPES-K1 LDC/KB/1 TinkerBell/KB/4 STANFORD/KB/3 hltcoe/KB/2 A2KD_Adept/KB/3 SAFT_ISI/KB/1 ICTCAS_OKN/KB/3 Precision Recall F1 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 ENG-LDC-MAX-ALL-Macro-HopALL-ALL-SLOT-TYPES-K1 LDC/KB/1 TinkerBell/KB/4 STANFORD/KB/3 A2KD_Adept/KB/1 hltcoe/KB/4 SAFT_ISI/KB/1 ICTCAS_OKN/KB/3 newbie_mr_01/KB/1 Precision Recall F1 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 SPA-LDC-MAX-ALL-Macro-Hop0-ALL-SLOT-TYPES-K1 LDC/KB/1 TinkerBell/KB/3 STANFORD/KB/2 SAFT_ISI/KB/1 Precision Recall F1 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 SPA-LDC-MAX-ALL-Macro-Hop1-ALL-SLOT-TYPES-K1 LDC/KB/1 STANFORD/KB/2 TinkerBell/KB/3 Precision Recall F1 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 SPA-LDC-MAX-ALL-Macro-HopALL-ALL-SLOT-TYPES-K1 LDC/KB/1 TinkerBell/KB/3 STANFORD/KB/2 SAFT_ISI/KB/1 Precision Recall F1 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 CMN-LDC-MAX-ALL-Macro-Hop0-ALL-SLOT-TYPES-K1 LDC/KB/1 TinkerBell/KB/2 A2KD_Adept/KB/4 ISCAS_Sogou/KB/4 STANFORD/KB/1 hltcoe/KB/2 SAFT_ISI/KB/1 Precision Recall F1 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 CMN-LDC-MAX-ALL-Macro-Hop1-ALL-SLOT-TYPES-K1 LDC/KB/1 TinkerBell/KB/4 ISCAS_Sogou/KB/1 STANFORD/KB/1 SAFT_ISI/KB/1 A2KD_Adept/KB/4 hltcoe/KB/2 Precision Recall F1 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 CMN-LDC-MAX-ALL-Macro-HopALL-ALL-SLOT-TYPES-K1 LDC/KB/1 TinkerBell/KB/2 ISCAS_Sogou/KB/4 A2KD_Adept/KB/4 STANFORD/KB/1 hltcoe/KB/2 SAFT_ISI/KB/1 Precision Recall F1 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 XLING-LDC-MAX-ALL-Macro-Hop0-ALL-SLOT-TYPES-K1 LDC/KB/1 TinkerBell/KB/5 STANFORD/KB/1 A2KD_Adept/KB/2 hltcoe/KB/2 Precision Recall F1 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 XLING-LDC-MAX-ALL-Macro-Hop1-ALL-SLOT-TYPES-K1 LDC/KB/1 TinkerBell/KB/4 STANFORD/KB/1 hltcoe/KB/3 A2KD_Adept/KB/2 Precision Recall F1 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 XLING-LDC-MAX-ALL-Macro-HopALL-ALL-SLOT-TYPES-K1 LDC/KB/1 TinkerBell/KB/4 STANFORD/KB/1 hltcoe/KB/2 A2KD_Adept/KB/2 Precision Recall F1

Hoa Dang, Shahzad Rajput (NIST) KBP 2017 Cold Start TAC 2017 32 / 48

HOP—0 HOP—1 HOP—ALL

slide-33
SLIDE 33

SF/KB Results

K1 - LDC-MEAN-ALL-Macro - AP (ALL-SLOT-TYPES)

0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 ENG-LDC-MEAN-ALL-Macro-Hop0-AP-ALL-SLOT-TYPES-K1 LDC/KB/1 TinkerBell/KB/4 STANFORD/KB/3 A2KD_Adept/KB/3 hltcoe/KB/4 SAFT_ISI/KB/1 ICTCAS_OKN/KB/3 newbie_mr_01/KB/1 AP 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 ENG-LDC-MEAN-ALL-Macro-Hop1-AP-ALL-SLOT-TYPES-K1 LDC/KB/1 TinkerBell/KB/4 STANFORD/KB/1 A2KD_Adept/KB/3 hltcoe/KB/4 SAFT_ISI/KB/1 ICTCAS_OKN/KB/3 AP 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 ENG-LDC-MEAN-ALL-Macro-HopALL-AP-ALL-SLOT-TYPES-K1 LDC/KB/1 A2KD_Adept/KB/3 TinkerBell/KB/4 STANFORD/KB/3 hltcoe/KB/4 SAFT_ISI/KB/1 ICTCAS_OKN/KB/3 newbie_mr_01/KB/1 AP 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 SPA-LDC-MEAN-ALL-Macro-Hop0-AP-ALL-SLOT-TYPES-K1 LDC/KB/1 TinkerBell/KB/3 STANFORD/KB/2 SAFT_ISI/KB/1 AP 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 SPA-LDC-MEAN-ALL-Macro-Hop1-AP-ALL-SLOT-TYPES-K1 LDC/KB/1 STANFORD/KB/2 TinkerBell/KB/3 AP 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 SPA-LDC-MEAN-ALL-Macro-HopALL-AP-ALL-SLOT-TYPES-K1 LDC/KB/1 TinkerBell/KB/3 STANFORD/KB/2 SAFT_ISI/KB/1 AP 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 CMN-LDC-MEAN-ALL-Macro-Hop0-AP-ALL-SLOT-TYPES-K1 LDC/KB/1 TinkerBell/KB/2 A2KD_Adept/KB/4 STANFORD/KB/1 ISCAS_Sogou/KB/4 hltcoe/KB/2 SAFT_ISI/KB/1 AP 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 CMN-LDC-MEAN-ALL-Macro-Hop1-AP-ALL-SLOT-TYPES-K1 LDC/KB/1 TinkerBell/KB/2 STANFORD/KB/1 ISCAS_Sogou/KB/1 A2KD_Adept/KB/2 hltcoe/KB/2 SAFT_ISI/KB/1 AP 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 CMN-LDC-MEAN-ALL-Macro-HopALL-AP-ALL-SLOT-TYPES-K1 LDC/KB/1 TinkerBell/KB/2 A2KD_Adept/KB/4 ISCAS_Sogou/KB/4 STANFORD/KB/1 hltcoe/KB/2 SAFT_ISI/KB/1 AP 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 XLING-LDC-MEAN-ALL-Macro-Hop0-AP-ALL-SLOT-TYPES-K1 LDC/KB/1 TinkerBell/KB/4 STANFORD/KB/1 A2KD_Adept/KB/2 hltcoe/KB/1 AP 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 XLING-LDC-MEAN-ALL-Macro-Hop1-AP-ALL-SLOT-TYPES-K1 LDC/KB/1 TinkerBell/KB/4 STANFORD/KB/1 hltcoe/KB/3 A2KD_Adept/KB/2 AP 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 XLING-LDC-MEAN-ALL-Macro-HopALL-AP-ALL-SLOT-TYPES-K1 LDC/KB/1 TinkerBell/KB/4 STANFORD/KB/1 A2KD_Adept/KB/2 hltcoe/KB/1 AP

Hoa Dang, Shahzad Rajput (NIST) KBP 2017 Cold Start TAC 2017 33 / 48

slide-34
SLIDE 34

SF/KB Results

K1 - LDC-MEAN-ALL-Macro - AP (EVENT-SLOTS-only)

0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 ENG-LDC-MEAN-ALL-Macro-Hop0-AP-EVENT-SLOTS-only-K1 LDC/KB/1 A2KD_Adept/KB/1 SAFT_ISI/KB/1 TinkerBell/KB/2 AP 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 ENG-LDC-MEAN-ALL-Macro-HopALL-AP-EVENT-SLOTS-only-K1 LDC/KB/1 A2KD_Adept/KB/1 SAFT_ISI/KB/1 TinkerBell/KB/2 AP 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 SPA-LDC-MEAN-ALL-Macro-Hop0-AP-EVENT-SLOTS-only-K1 LDC/KB/1 SAFT_ISI/KB/1 TinkerBell/KB/5 AP 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 SPA-LDC-MEAN-ALL-Macro-HopALL-AP-EVENT-SLOTS-only-K1 LDC/KB/1 SAFT_ISI/KB/1 TinkerBell/KB/5 AP 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 CMN-LDC-MEAN-ALL-Macro-Hop0-AP-EVENT-SLOTS-only-K1 LDC/KB/1 TinkerBell/KB/1 A2KD_Adept/KB/3 ISCAS_Sogou/KB/2 AP 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 CMN-LDC-MEAN-ALL-Macro-HopALL-AP-EVENT-SLOTS-only-K1 LDC/KB/1 TinkerBell/KB/1 A2KD_Adept/KB/3 ISCAS_Sogou/KB/2 AP 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 XLING-LDC-MEAN-ALL-Macro-Hop0-AP-EVENT-SLOTS-only-K1 LDC/KB/1 A2KD_Adept/KB/1 TinkerBell/KB/3 AP 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 XLING-LDC-MEAN-ALL-Macro-HopALL-AP-EVENT-SLOTS-only-K1 LDC/KB/1 A2KD_Adept/KB/1 TinkerBell/KB/3 AP

Hoa Dang, Shahzad Rajput (NIST) KBP 2017 Cold Start TAC 2017 34 / 48

slide-35
SLIDE 35

SF/KB Results

K1 - LDC-MEAN-ALL-Macro - AP (SENTIMENT-SLOTS-only)

0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 ENG-LDC-MEAN-ALL-Macro-Hop0-AP-SENTIMENT-SLOTS-only-K1 LDC/KB/1 TinkerBell/KB/1 ICTCAS_OKN/KB/3 SAFT_ISI/KB/1 AP 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 ENG-LDC-MEAN-ALL-Macro-Hop1-AP-SENTIMENT-SLOTS-only-K1 LDC/KB/1 TinkerBell/KB/3 AP 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 ENG-LDC-MEAN-ALL-Macro-HopALL-AP-SENTIMENT-SLOTS-only-K1 LDC/KB/1 TinkerBell/KB/3 ICTCAS_OKN/KB/3 SAFT_ISI/KB/1 AP 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 SPA-LDC-MEAN-ALL-Macro-Hop0-AP-SENTIMENT-SLOTS-only-K1 LDC/KB/1 TinkerBell/KB/3 SAFT_ISI/KB/1 AP 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 SPA-LDC-MEAN-ALL-Macro-Hop1-AP-SENTIMENT-SLOTS-only-K1 LDC/KB/1 AP 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 SPA-LDC-MEAN-ALL-Macro-HopALL-AP-SENTIMENT-SLOTS-only-K1 LDC/KB/1 TinkerBell/KB/3 SAFT_ISI/KB/1 AP 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 CMN-LDC-MEAN-ALL-Macro-Hop0-AP-SENTIMENT-SLOTS-only-K1 LDC/KB/1 TinkerBell/KB/2 ISCAS_Sogou/KB/4 SAFT_ISI/KB/1 AP 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 CMN-LDC-MEAN-ALL-Macro-Hop1-AP-SENTIMENT-SLOTS-only-K1 LDC/KB/1 TinkerBell/KB/2 ISCAS_Sogou/KB/5 SAFT_ISI/KB/1 AP 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 CMN-LDC-MEAN-ALL-Macro-HopALL-AP-SENTIMENT-SLOTS-only-K1 LDC/KB/1 TinkerBell/KB/2 ISCAS_Sogou/KB/4 SAFT_ISI/KB/1 AP 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 XLING-LDC-MEAN-ALL-Macro-Hop0-AP-SENTIMENT-SLOTS-only-K1 LDC/KB/1 TinkerBell/KB/3 AP 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 XLING-LDC-MEAN-ALL-Macro-Hop1-AP-SENTIMENT-SLOTS-only-K1 LDC/KB/1 TinkerBell/KB/1 AP 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 XLING-LDC-MEAN-ALL-Macro-HopALL-AP-SENTIMENT-SLOTS-only-K1 LDC/KB/1 TinkerBell/KB/3 AP

Hoa Dang, Shahzad Rajput (NIST) KBP 2017 Cold Start TAC 2017 35 / 48

slide-36
SLIDE 36

SF/KB Results

K1 - LDC-MEAN-ALL-Macro - AP (SF-SLOTS-only)

0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 ENG-LDC-MEAN-ALL-Macro-Hop0-AP-SF-SLOTS-only-K1 LDC/KB/1 STANFORD/KB/3 TinkerBell/KB/4 hltcoe/KB/4 UNIST_SAIL/SF/1 A2KD_Adept/KB/3 Y_dcd_zju/SF/2 SAFT_ISI/KB/1 ICTCAS_OKN/KB/3 dws_uma/SF/1 newbie_mr_01/KB/1 AP 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 ENG-LDC-MEAN-ALL-Macro-Hop1-AP-SF-SLOTS-only-K1 LDC/KB/1 TinkerBell/KB/4 STANFORD/KB/1 A2KD_Adept/KB/4 hltcoe/KB/2 dws_uma/SF/1 UNIST_SAIL/SF/1 Y_dcd_zju/SF/2 SAFT_ISI/KB/1 ICTCAS_OKN/KB/3 AP 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 ENG-LDC-MEAN-ALL-Macro-HopALL-AP-SF-SLOTS-only-K1 LDC/KB/1 STANFORD/KB/3 TinkerBell/KB/4 hltcoe/KB/4 UNIST_SAIL/SF/1 A2KD_Adept/KB/4 Y_dcd_zju/SF/2 SAFT_ISI/KB/1 ICTCAS_OKN/KB/3 dws_uma/SF/1 newbie_mr_01/KB/1 AP 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 SPA-LDC-MEAN-ALL-Macro-Hop0-AP-SF-SLOTS-only-K1 LDC/KB/1 STANFORD/KB/2 TinkerBell/KB/4 AP 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 SPA-LDC-MEAN-ALL-Macro-Hop1-AP-SF-SLOTS-only-K1 LDC/KB/1 STANFORD/KB/2 TinkerBell/KB/4 AP 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 SPA-LDC-MEAN-ALL-Macro-HopALL-AP-SF-SLOTS-only-K1 LDC/KB/1 STANFORD/KB/2 TinkerBell/KB/4 AP 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 CMN-LDC-MEAN-ALL-Macro-Hop0-AP-SF-SLOTS-only-K1 LDC/KB/1 STANFORD/KB/1 TinkerBell/KB/2 A2KD_Adept/KB/4 hltcoe/KB/2 ISCAS_Sogou/KB/3 AP 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 CMN-LDC-MEAN-ALL-Macro-Hop1-AP-SF-SLOTS-only-K1 LDC/KB/1 STANFORD/KB/1 TinkerBell/KB/1 ISCAS_Sogou/KB/1 hltcoe/KB/2 A2KD_Adept/KB/2 AP 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 CMN-LDC-MEAN-ALL-Macro-HopALL-AP-SF-SLOTS-only-K1 LDC/KB/1 STANFORD/KB/1 TinkerBell/KB/2 A2KD_Adept/KB/4 hltcoe/KB/2 ISCAS_Sogou/KB/3 AP 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 XLING-LDC-MEAN-ALL-Macro-Hop0-AP-SF-SLOTS-only-K1 LDC/KB/1 TinkerBell/KB/4 STANFORD/KB/1 hltcoe/KB/1 A2KD_Adept/KB/2 AP 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 XLING-LDC-MEAN-ALL-Macro-Hop1-AP-SF-SLOTS-only-K1 LDC/KB/1 TinkerBell/KB/4 STANFORD/KB/1 hltcoe/KB/1 A2KD_Adept/KB/2 AP 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 XLING-LDC-MEAN-ALL-Macro-HopALL-AP-SF-SLOTS-only-K1 LDC/KB/1 TinkerBell/KB/4 STANFORD/KB/1 hltcoe/KB/1 A2KD_Adept/KB/2 AP

Hoa Dang, Shahzad Rajput (NIST) KBP 2017 Cold Start TAC 2017 36 / 48

slide-37
SLIDE 37

SF/KB Results

K1 - LDC-MEAN-ALL-Macro - F1 (ALL-SLOT-TYPES)

0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 ENG-LDC-MEAN-ALL-Macro-Hop0-ALL-SLOT-TYPES-K1 LDC/KB/1 A2KD_Adept/KB/2 TinkerBell/KB/4 STANFORD/KB/3 hltcoe/KB/4 SAFT_ISI/KB/1 ICTCAS_OKN/KB/3 newbie_mr_01/KB/1 Precision Recall F1 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 ENG-LDC-MEAN-ALL-Macro-Hop1-ALL-SLOT-TYPES-K1 LDC/KB/1 TinkerBell/KB/4 STANFORD/KB/1 hltcoe/KB/4 A2KD_Adept/KB/3 SAFT_ISI/KB/1 ICTCAS_OKN/KB/3 Precision Recall F1 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 ENG-LDC-MEAN-ALL-Macro-HopALL-ALL-SLOT-TYPES-K1 LDC/KB/1 TinkerBell/KB/4 STANFORD/KB/3 A2KD_Adept/KB/1 hltcoe/KB/4 SAFT_ISI/KB/1 ICTCAS_OKN/KB/3 newbie_mr_01/KB/1 Precision Recall F1 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 SPA-LDC-MEAN-ALL-Macro-Hop0-ALL-SLOT-TYPES-K1 LDC/KB/1 TinkerBell/KB/3 STANFORD/KB/2 SAFT_ISI/KB/1 Precision Recall F1 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 SPA-LDC-MEAN-ALL-Macro-Hop1-ALL-SLOT-TYPES-K1 LDC/KB/1 STANFORD/KB/2 TinkerBell/KB/3 Precision Recall F1 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 SPA-LDC-MEAN-ALL-Macro-HopALL-ALL-SLOT-TYPES-K1 LDC/KB/1 TinkerBell/KB/3 STANFORD/KB/2 SAFT_ISI/KB/1 Precision Recall F1 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 CMN-LDC-MEAN-ALL-Macro-Hop0-ALL-SLOT-TYPES-K1 LDC/KB/1 TinkerBell/KB/2 A2KD_Adept/KB/4 ISCAS_Sogou/KB/1 STANFORD/KB/1 hltcoe/KB/2 SAFT_ISI/KB/1 Precision Recall F1 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 CMN-LDC-MEAN-ALL-Macro-Hop1-ALL-SLOT-TYPES-K1 LDC/KB/1 TinkerBell/KB/4 SAFT_ISI/KB/1 STANFORD/KB/1 ISCAS_Sogou/KB/1 A2KD_Adept/KB/4 hltcoe/KB/2 Precision Recall F1 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 CMN-LDC-MEAN-ALL-Macro-HopALL-ALL-SLOT-TYPES-K1 LDC/KB/1 TinkerBell/KB/2 A2KD_Adept/KB/4 ISCAS_Sogou/KB/1 STANFORD/KB/1 hltcoe/KB/2 SAFT_ISI/KB/1 Precision Recall F1 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 XLING-LDC-MEAN-ALL-Macro-Hop0-ALL-SLOT-TYPES-K1 LDC/KB/1 TinkerBell/KB/4 STANFORD/KB/1 A2KD_Adept/KB/2 hltcoe/KB/1 Precision Recall F1 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 XLING-LDC-MEAN-ALL-Macro-Hop1-ALL-SLOT-TYPES-K1 LDC/KB/1 TinkerBell/KB/4 STANFORD/KB/1 hltcoe/KB/3 A2KD_Adept/KB/2 Precision Recall F1 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 XLING-LDC-MEAN-ALL-Macro-HopALL-ALL-SLOT-TYPES-K1 LDC/KB/1 TinkerBell/KB/4 STANFORD/KB/1 hltcoe/KB/1 A2KD_Adept/KB/2 Precision Recall F1

Hoa Dang, Shahzad Rajput (NIST) KBP 2017 Cold Start TAC 2017 37 / 48

slide-38
SLIDE 38

SF/KB Results

K1 - LDC-MEAN-ALL-Macro - F1 (EVENT-SLOTS-only)

0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 ENG-LDC-MEAN-ALL-Macro-Hop0-EVENT-SLOTS-only-K1 LDC/KB/1 A2KD_Adept/KB/1 SAFT_ISI/KB/1 TinkerBell/KB/2 Precision Recall F1 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 ENG-LDC-MEAN-ALL-Macro-HopALL-EVENT-SLOTS-only-K1 LDC/KB/1 A2KD_Adept/KB/1 SAFT_ISI/KB/1 TinkerBell/KB/2 Precision Recall F1 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 SPA-LDC-MEAN-ALL-Macro-Hop0-EVENT-SLOTS-only-K1 LDC/KB/1 SAFT_ISI/KB/1 TinkerBell/KB/5 Precision Recall F1 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 SPA-LDC-MEAN-ALL-Macro-HopALL-EVENT-SLOTS-only-K1 LDC/KB/1 SAFT_ISI/KB/1 TinkerBell/KB/5 Precision Recall F1 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 CMN-LDC-MEAN-ALL-Macro-Hop0-EVENT-SLOTS-only-K1 LDC/KB/1 TinkerBell/KB/1 A2KD_Adept/KB/4 ISCAS_Sogou/KB/2 Precision Recall F1 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 CMN-LDC-MEAN-ALL-Macro-HopALL-EVENT-SLOTS-only-K1 LDC/KB/1 TinkerBell/KB/1 A2KD_Adept/KB/4 ISCAS_Sogou/KB/2 Precision Recall F1 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 XLING-LDC-MEAN-ALL-Macro-Hop0-EVENT-SLOTS-only-K1 LDC/KB/1 A2KD_Adept/KB/1 TinkerBell/KB/3 Precision Recall F1 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 XLING-LDC-MEAN-ALL-Macro-HopALL-EVENT-SLOTS-only-K1 LDC/KB/1 A2KD_Adept/KB/1 TinkerBell/KB/3 Precision Recall F1

Hoa Dang, Shahzad Rajput (NIST) KBP 2017 Cold Start TAC 2017 38 / 48

slide-39
SLIDE 39

SF/KB Results

K1 - LDC-MEAN-ALL-Macro - F1 (SENTIMENT-SLOTS-only)

0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 ENG-LDC-MEAN-ALL-Macro-Hop0-SENTIMENT-SLOTS-only-K1 LDC/KB/1 TinkerBell/KB/1 ICTCAS_OKN/KB/3 SAFT_ISI/KB/1 Precision Recall F1 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 ENG-LDC-MEAN-ALL-Macro-Hop1-SENTIMENT-SLOTS-only-K1 LDC/KB/1 TinkerBell/KB/1 Precision Recall F1 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 ENG-LDC-MEAN-ALL-Macro-HopALL-SENTIMENT-SLOTS-only-K1 LDC/KB/1 TinkerBell/KB/3 ICTCAS_OKN/KB/3 SAFT_ISI/KB/1 Precision Recall F1 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 SPA-LDC-MEAN-ALL-Macro-Hop0-SENTIMENT-SLOTS-only-K1 LDC/KB/1 TinkerBell/KB/3 SAFT_ISI/KB/1 Precision Recall F1 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 SPA-LDC-MEAN-ALL-Macro-Hop1-SENTIMENT-SLOTS-only-K1 LDC/KB/1 TinkerBell/KB/3 SAFT_ISI/KB/1 Precision Recall F1 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 SPA-LDC-MEAN-ALL-Macro-HopALL-SENTIMENT-SLOTS-only-K1 LDC/KB/1 TinkerBell/KB/3 SAFT_ISI/KB/1 Precision Recall F1 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 CMN-LDC-MEAN-ALL-Macro-Hop0-SENTIMENT-SLOTS-only-K1 LDC/KB/1 TinkerBell/KB/2 ISCAS_Sogou/KB/4 SAFT_ISI/KB/1 Precision Recall F1 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 CMN-LDC-MEAN-ALL-Macro-Hop1-SENTIMENT-SLOTS-only-K1 TinkerBell/KB/4 LDC/KB/1 SAFT_ISI/KB/1 ISCAS_Sogou/KB/4 Precision Recall F1 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 CMN-LDC-MEAN-ALL-Macro-HopALL-SENTIMENT-SLOTS-only-K1 LDC/KB/1 TinkerBell/KB/2 ISCAS_Sogou/KB/4 SAFT_ISI/KB/1 Precision Recall F1 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 XLING-LDC-MEAN-ALL-Macro-Hop0-SENTIMENT-SLOTS-only-K1 LDC/KB/1 TinkerBell/KB/3 Precision Recall F1 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 XLING-LDC-MEAN-ALL-Macro-Hop1-SENTIMENT-SLOTS-only-K1 TinkerBell/KB/1 LDC/KB/1 Precision Recall F1 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 XLING-LDC-MEAN-ALL-Macro-HopALL-SENTIMENT-SLOTS-only-K1 LDC/KB/1 TinkerBell/KB/3 Precision Recall F1

Hoa Dang, Shahzad Rajput (NIST) KBP 2017 Cold Start TAC 2017 39 / 48

slide-40
SLIDE 40

SF/KB Results

K1 - LDC-MEAN-ALL-Macro - F1 (SF-SLOTS-only)

0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 ENG-LDC-MEAN-ALL-Macro-Hop0-SF-SLOTS-only-K1 LDC/KB/1 STANFORD/KB/3 TinkerBell/KB/4 hltcoe/KB/4 UNIST_SAIL/SF/2 A2KD_Adept/KB/3 Y_dcd_zju/SF/1 ICTCAS_OKN/KB/3 SAFT_ISI/KB/1 dws_uma/SF/1 newbie_mr_01/KB/1 Precision Recall F1 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 ENG-LDC-MEAN-ALL-Macro-Hop1-SF-SLOTS-only-K1 TinkerBell/KB/4 STANFORD/KB/1 LDC/KB/1 hltcoe/KB/4 A2KD_Adept/KB/3 UNIST_SAIL/SF/1 dws_uma/SF/1 Y_dcd_zju/SF/2 SAFT_ISI/KB/1 ICTCAS_OKN/KB/3 Precision Recall F1 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 ENG-LDC-MEAN-ALL-Macro-HopALL-SF-SLOTS-only-K1 LDC/KB/1 STANFORD/KB/3 TinkerBell/KB/4 hltcoe/KB/4 UNIST_SAIL/SF/2 A2KD_Adept/KB/3 Y_dcd_zju/SF/2 ICTCAS_OKN/KB/3 SAFT_ISI/KB/1 dws_uma/SF/1 newbie_mr_01/KB/1 Precision Recall F1 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 SPA-LDC-MEAN-ALL-Macro-Hop0-SF-SLOTS-only-K1 LDC/KB/1 STANFORD/KB/2 TinkerBell/KB/4 Precision Recall F1 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 SPA-LDC-MEAN-ALL-Macro-Hop1-SF-SLOTS-only-K1 LDC/KB/1 STANFORD/KB/2 TinkerBell/KB/4 Precision Recall F1 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 SPA-LDC-MEAN-ALL-Macro-HopALL-SF-SLOTS-only-K1 LDC/KB/1 STANFORD/KB/2 TinkerBell/KB/4 Precision Recall F1 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 CMN-LDC-MEAN-ALL-Macro-Hop0-SF-SLOTS-only-K1 LDC/KB/1 STANFORD/KB/1 TinkerBell/KB/2 A2KD_Adept/KB/4 hltcoe/KB/2 ISCAS_Sogou/KB/1 Precision Recall F1 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 CMN-LDC-MEAN-ALL-Macro-Hop1-SF-SLOTS-only-K1 LDC/KB/1 STANFORD/SF/4 TinkerBell/KB/2 A2KD_Adept/KB/4 ISCAS_Sogou/KB/1 hltcoe/KB/2 Precision Recall F1 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 CMN-LDC-MEAN-ALL-Macro-HopALL-SF-SLOTS-only-K1 LDC/KB/1 STANFORD/SF/4 TinkerBell/KB/2 A2KD_Adept/KB/4 hltcoe/KB/2 ISCAS_Sogou/KB/1 Precision Recall F1 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 XLING-LDC-MEAN-ALL-Macro-Hop0-SF-SLOTS-only-K1 LDC/KB/1 TinkerBell/KB/4 STANFORD/KB/1 hltcoe/KB/1 A2KD_Adept/KB/2 Precision Recall F1 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 XLING-LDC-MEAN-ALL-Macro-Hop1-SF-SLOTS-only-K1 TinkerBell/KB/4 LDC/KB/1 STANFORD/KB/1 hltcoe/KB/3 A2KD_Adept/KB/2 Precision Recall F1 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 XLING-LDC-MEAN-ALL-Macro-HopALL-SF-SLOTS-only-K1 LDC/KB/1 TinkerBell/KB/4 STANFORD/KB/1 hltcoe/KB/1 A2KD_Adept/KB/2 Precision Recall F1

Hoa Dang, Shahzad Rajput (NIST) KBP 2017 Cold Start TAC 2017 40 / 48

slide-41
SLIDE 41

SF/KB Results

K3 - LDC-MEAN-ALL-Macro - AP (ALL-SLOT-TYPES)

0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 ENG-LDC-MEAN-ALL-Macro-Hop0-AP-ALL-SLOT-TYPES-K3 LDC/KB/1 TinkerBell/KB/3 STANFORD/KB/3 hltcoe/KB/4 A2KD_Adept/KB/4 SAFT_ISI/KB/1 ICTCAS_OKN/KB/3 newbie_mr_01/KB/1 AP 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 ENG-LDC-MEAN-ALL-Macro-Hop1-AP-ALL-SLOT-TYPES-K3 LDC/KB/1 TinkerBell/KB/3 STANFORD/KB/3 A2KD_Adept/KB/4 hltcoe/KB/1 SAFT_ISI/KB/1 ICTCAS_OKN/KB/1 AP 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 ENG-LDC-MEAN-ALL-Macro-HopALL-AP-ALL-SLOT-TYPES-K3 LDC/KB/1 TinkerBell/KB/3 A2KD_Adept/KB/4 STANFORD/KB/1 hltcoe/KB/4 SAFT_ISI/KB/1 ICTCAS_OKN/KB/3 newbie_mr_01/KB/1 AP 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 SPA-LDC-MEAN-ALL-Macro-Hop0-AP-ALL-SLOT-TYPES-K3 LDC/KB/1 TinkerBell/KB/3 STANFORD/KB/1 SAFT_ISI/KB/1 AP 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 SPA-LDC-MEAN-ALL-Macro-Hop1-AP-ALL-SLOT-TYPES-K3 LDC/KB/1 STANFORD/KB/1 TinkerBell/KB/3 AP 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 SPA-LDC-MEAN-ALL-Macro-HopALL-AP-ALL-SLOT-TYPES-K3 LDC/KB/1 TinkerBell/KB/3 STANFORD/KB/1 SAFT_ISI/KB/1 AP 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 CMN-LDC-MEAN-ALL-Macro-Hop0-AP-ALL-SLOT-TYPES-K3 LDC/KB/1 TinkerBell/KB/5 A2KD_Adept/KB/4 ISCAS_Sogou/KB/4 STANFORD/KB/1 hltcoe/KB/1 SAFT_ISI/KB/1 AP 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 CMN-LDC-MEAN-ALL-Macro-Hop1-AP-ALL-SLOT-TYPES-K3 LDC/KB/1 TinkerBell/KB/5 ISCAS_Sogou/KB/1 A2KD_Adept/KB/2 STANFORD/KB/1 hltcoe/KB/1 SAFT_ISI/KB/1 AP 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 CMN-LDC-MEAN-ALL-Macro-HopALL-AP-ALL-SLOT-TYPES-K3 LDC/KB/1 TinkerBell/KB/5 A2KD_Adept/KB/4 ISCAS_Sogou/KB/4 STANFORD/KB/4 hltcoe/KB/3 SAFT_ISI/KB/1 AP 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 XLING-LDC-MEAN-ALL-Macro-Hop0-AP-ALL-SLOT-TYPES-K3 LDC/KB/1 TinkerBell/KB/3 STANFORD/KB/1 hltcoe/KB/1 A2KD_Adept/KB/2 AP 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 XLING-LDC-MEAN-ALL-Macro-Hop1-AP-ALL-SLOT-TYPES-K3 LDC/KB/1 TinkerBell/KB/3 STANFORD/KB/1 hltcoe/KB/1 A2KD_Adept/KB/1 AP 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 XLING-LDC-MEAN-ALL-Macro-HopALL-AP-ALL-SLOT-TYPES-K3 LDC/KB/1 TinkerBell/KB/3 A2KD_Adept/KB/2 STANFORD/KB/1 hltcoe/KB/1 AP

Hoa Dang, Shahzad Rajput (NIST) KBP 2017 Cold Start TAC 2017 41 / 48

slide-42
SLIDE 42

SF/KB Results

K3 - LDC-MEAN-ALL-Macro - AP (EVENT-SLOTS-only)

0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 ENG-LDC-MEAN-ALL-Macro-Hop0-AP-EVENT-SLOTS-only-K3 LDC/KB/1 A2KD_Adept/KB/1 SAFT_ISI/KB/1 TinkerBell/KB/3 AP 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 ENG-LDC-MEAN-ALL-Macro-HopALL-AP-EVENT-SLOTS-only-K3 LDC/KB/1 A2KD_Adept/KB/1 SAFT_ISI/KB/1 TinkerBell/KB/3 AP 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 SPA-LDC-MEAN-ALL-Macro-Hop0-AP-EVENT-SLOTS-only-K3 LDC/KB/1 SAFT_ISI/KB/1 TinkerBell/KB/1 AP 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 SPA-LDC-MEAN-ALL-Macro-HopALL-AP-EVENT-SLOTS-only-K3 LDC/KB/1 SAFT_ISI/KB/1 TinkerBell/KB/1 AP 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 CMN-LDC-MEAN-ALL-Macro-Hop0-AP-EVENT-SLOTS-only-K3 LDC/KB/1 TinkerBell/KB/5 A2KD_Adept/KB/3 ISCAS_Sogou/KB/2 AP 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 CMN-LDC-MEAN-ALL-Macro-HopALL-AP-EVENT-SLOTS-only-K3 LDC/KB/1 TinkerBell/KB/5 A2KD_Adept/KB/3 ISCAS_Sogou/KB/2 AP 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 XLING-LDC-MEAN-ALL-Macro-Hop0-AP-EVENT-SLOTS-only-K3 LDC/KB/1 A2KD_Adept/KB/1 TinkerBell/KB/3 AP 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 XLING-LDC-MEAN-ALL-Macro-HopALL-AP-EVENT-SLOTS-only-K3 LDC/KB/1 A2KD_Adept/KB/1 TinkerBell/KB/3 AP

Hoa Dang, Shahzad Rajput (NIST) KBP 2017 Cold Start TAC 2017 42 / 48

slide-43
SLIDE 43

SF/KB Results

K3 - LDC-MEAN-ALL-Macro - AP (SENTIMENT-SLOTS-only)

0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 ENG-LDC-MEAN-ALL-Macro-Hop0-AP-SENTIMENT-SLOTS-only-K3 LDC/KB/1 TinkerBell/KB/3 ICTCAS_OKN/KB/3 SAFT_ISI/KB/1 AP 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 ENG-LDC-MEAN-ALL-Macro-Hop1-AP-SENTIMENT-SLOTS-only-K3 LDC/KB/1 TinkerBell/KB/3 AP 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 ENG-LDC-MEAN-ALL-Macro-HopALL-AP-SENTIMENT-SLOTS-only-K3 LDC/KB/1 TinkerBell/KB/3 ICTCAS_OKN/KB/3 SAFT_ISI/KB/1 AP 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 SPA-LDC-MEAN-ALL-Macro-Hop0-AP-SENTIMENT-SLOTS-only-K3 LDC/KB/1 TinkerBell/KB/3 SAFT_ISI/KB/1 AP 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 SPA-LDC-MEAN-ALL-Macro-Hop1-AP-SENTIMENT-SLOTS-only-K3 LDC/KB/1 AP 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 SPA-LDC-MEAN-ALL-Macro-HopALL-AP-SENTIMENT-SLOTS-only-K3 LDC/KB/1 TinkerBell/KB/3 SAFT_ISI/KB/1 AP 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 CMN-LDC-MEAN-ALL-Macro-Hop0-AP-SENTIMENT-SLOTS-only-K3 LDC/KB/1 TinkerBell/KB/2 ISCAS_Sogou/KB/4 SAFT_ISI/KB/1 AP 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 CMN-LDC-MEAN-ALL-Macro-Hop1-AP-SENTIMENT-SLOTS-only-K3 LDC/KB/1 TinkerBell/KB/2 ISCAS_Sogou/KB/4 SAFT_ISI/KB/1 AP 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 CMN-LDC-MEAN-ALL-Macro-HopALL-AP-SENTIMENT-SLOTS-only-K3 LDC/KB/1 TinkerBell/KB/2 ISCAS_Sogou/KB/1 SAFT_ISI/KB/1 AP 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 XLING-LDC-MEAN-ALL-Macro-Hop0-AP-SENTIMENT-SLOTS-only-K3 LDC/KB/1 TinkerBell/KB/3 AP 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 XLING-LDC-MEAN-ALL-Macro-Hop1-AP-SENTIMENT-SLOTS-only-K3 LDC/KB/1 TinkerBell/KB/2 AP 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 XLING-LDC-MEAN-ALL-Macro-HopALL-AP-SENTIMENT-SLOTS-only-K3 LDC/KB/1 TinkerBell/KB/3 AP

Hoa Dang, Shahzad Rajput (NIST) KBP 2017 Cold Start TAC 2017 43 / 48

slide-44
SLIDE 44

SF/KB Results

K3 - LDC-MEAN-ALL-Macro - AP (SF-SLOTS-only)

0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 ENG-LDC-MEAN-ALL-Macro-Hop0-AP-SF-SLOTS-only-K3 LDC/KB/1 STANFORD/KB/3 TinkerBell/KB/3 hltcoe/KB/4 A2KD_Adept/KB/4 SAFT_ISI/KB/1 ICTCAS_OKN/KB/3 newbie_mr_01/KB/1 AP 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 ENG-LDC-MEAN-ALL-Macro-Hop1-AP-SF-SLOTS-only-K3 LDC/KB/1 STANFORD/KB/3 TinkerBell/KB/3 A2KD_Adept/KB/4 hltcoe/KB/1 SAFT_ISI/KB/1 ICTCAS_OKN/KB/3 AP 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 ENG-LDC-MEAN-ALL-Macro-HopALL-AP-SF-SLOTS-only-K3 LDC/KB/1 STANFORD/KB/3 hltcoe/KB/4 TinkerBell/KB/2 A2KD_Adept/KB/4 SAFT_ISI/KB/1 ICTCAS_OKN/KB/3 newbie_mr_01/KB/1 AP 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 SPA-LDC-MEAN-ALL-Macro-Hop0-AP-SF-SLOTS-only-K3 LDC/KB/1 STANFORD/KB/2 TinkerBell/KB/1 AP 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 SPA-LDC-MEAN-ALL-Macro-Hop1-AP-SF-SLOTS-only-K3 LDC/KB/1 STANFORD/KB/2 TinkerBell/KB/1 AP 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 SPA-LDC-MEAN-ALL-Macro-HopALL-AP-SF-SLOTS-only-K3 LDC/KB/1 STANFORD/KB/2 TinkerBell/KB/1 AP 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 CMN-LDC-MEAN-ALL-Macro-Hop0-AP-SF-SLOTS-only-K3 LDC/KB/1 STANFORD/KB/1 TinkerBell/KB/5 A2KD_Adept/KB/4 hltcoe/KB/2 ISCAS_Sogou/KB/3 AP 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 CMN-LDC-MEAN-ALL-Macro-Hop1-AP-SF-SLOTS-only-K3 LDC/KB/1 TinkerBell/KB/5 STANFORD/KB/1 A2KD_Adept/KB/2 ISCAS_Sogou/KB/3 hltcoe/KB/1 AP 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 CMN-LDC-MEAN-ALL-Macro-HopALL-AP-SF-SLOTS-only-K3 LDC/KB/1 STANFORD/KB/4 A2KD_Adept/KB/4 TinkerBell/KB/3 hltcoe/KB/3 ISCAS_Sogou/KB/3 AP 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 XLING-LDC-MEAN-ALL-Macro-Hop0-AP-SF-SLOTS-only-K3 LDC/KB/1 TinkerBell/KB/3 STANFORD/KB/1 hltcoe/KB/1 A2KD_Adept/KB/2 AP 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 XLING-LDC-MEAN-ALL-Macro-Hop1-AP-SF-SLOTS-only-K3 LDC/KB/1 TinkerBell/KB/3 STANFORD/KB/1 hltcoe/KB/1 A2KD_Adept/KB/1 AP 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 XLING-LDC-MEAN-ALL-Macro-HopALL-AP-SF-SLOTS-only-K3 LDC/KB/1 TinkerBell/KB/3 STANFORD/KB/1 hltcoe/KB/1 A2KD_Adept/KB/2 AP

Hoa Dang, Shahzad Rajput (NIST) KBP 2017 Cold Start TAC 2017 44 / 48

slide-45
SLIDE 45

Correlation between AP and F1

0.02 0.04 0.06 0.08 0.1 0.12 0.14 0.16 0.18 0.2 K3-AP 0.02 0.04 0.06 0.08 0.1 0.12 0.14 0.16 0.18 0.2 K1-F1 ALL-SLOT-TYPES - HopALL - LDC-MEAN-ALL-Macro ρ: 0.9897 τ: 0.7796

Hoa Dang, Shahzad Rajput (NIST) KBP 2017 Cold Start TAC 2017 45 / 48

slide-46
SLIDE 46

Correlation between AP and F1

0.05 0.1 0.15 0.2 0.25 0.3 K3-AP 0.05 0.1 0.15 0.2 0.25 0.3 K1-F1 SF-SLOTS-only - Hop0 - LDC-MEAN-ALL-Macro ρ: 0.9737 τ: 0.8531

Hoa Dang, Shahzad Rajput (NIST) KBP 2017 Cold Start TAC 2017 46 / 48

slide-47
SLIDE 47

Outline

1

Introduction Task Variants Changes in 2017

2

SF/KB Evaluation Definitions Queries Participants Results

3

Conclusion

Hoa Dang, Shahzad Rajput (NIST) KBP 2017 Cold Start TAC 2017 47 / 48

slide-48
SLIDE 48

Conclusion

AP scores have near perfect correlation with that of F scores Generally automatic systems appear to get about 70% or less performance as compared to manual systems. In some cases, we see automatic systems perform better manual system Significantly more number of KB submissions as compared to SF submissions Much room for improvement on Hop-1 performance Much room for improvement on event- and sentiment-based queries

Hoa Dang, Shahzad Rajput (NIST) KBP 2017 Cold Start TAC 2017 48 / 48