SLIDE 1
Unsupervised Coreference Resolution in a Nonparametric Bayesian - - PowerPoint PPT Presentation
Unsupervised Coreference Resolution in a Nonparametric Bayesian - - PowerPoint PPT Presentation
Unsupervised Coreference Resolution in a Nonparametric Bayesian Model Aria Haghighi and Dan Klein Computer Science Division University of California Berkeley Coreference Resolution The Weir Group , whose headquarters is in the U.S , is a
SLIDE 2
SLIDE 3
Coreference Resolution
The Weir Group , whose headquarters is in the U.S , is a large specialized corporation . This power plant ,which , will be situated in Jiangsu , has a large generation capacity.
SLIDE 4
Coreference Resolution
The Weir Group , whose headquarters is in the U.S , is a large specialized corporation . This power plant ,which , will be situated in Jiangsu , has a large generation capacity.
SLIDE 5
Coreference Resolution
.............. .......... .......... ... . ... ... . . ....... .......
Generative Story
SLIDE 6
Coreference Resolution
.............. .......... .......... ... . ... ... . . ....... .......
SLIDE 7
Coreference Resolution
Weir Group Weir HQ United States Weir Group Weir Group Weir Plant Weir Plant Jiangsu .............. .......... .......... ... . ... ... . . ....... .......
SLIDE 8
Coreference Resolution
Weir Group Weir HQ United States Weir Group Weir Group Weir Plant Weir Plant Jiangsu .............. .......... .......... ... . ... ... . . ....... .......
Entity
SLIDE 9
Coreference Resolution
Weir Group Weir HQ United States Weir Group Weir Group Weir Plant Weir Plant Jiangsu .............. .......... .......... ... . ... ... . . ....... .......
SLIDE 10
Coreference Resolution
“Weir group” “whose” “headquarters” “U.S” “corporation” “power plant” “which” “Jiangsu” Weir Group Weir HQ United States Weir Group Weir Group Weir Plant Weir Plant Jiangsu .............. .......... .......... ... . ... ... . . ....... .......
SLIDE 11
Coreference Resolution
“Weir group” “whose” “headquarters” “U.S” “corporation” “power plant” “which” “Jiangsu” Weir Group Weir HQ United States Weir Group Weir Group Weir Plant Weir Plant Jiangsu .............. .......... .......... ... . ... ... . . ....... .......
Mention
SLIDE 12
Coreference Resolution
“Weir group” “whose” “headquarters” “U.S” “corporation” “power plant” “which” “Jiangsu” Weir Group Weir HQ United States Weir Group Weir Group Weir Plant Weir Plant Jiangsu .............. .......... .......... ... . ... ... . . ....... .......
SLIDE 13
Coreference Resolution
.............. .......... .......... ... . ... ... . . ....... .......
Inference Time
SLIDE 14
Coreference Resolution
.............. .......... .......... ... . ... ... . . ....... .......
SLIDE 15
Coreference Resolution
“Weir group” “whose” “headquarters” “U.S” “corporation” “power plant” “which” “Jiangsu” .............. .......... .......... ... . ... ... . . ....... .......
SLIDE 16
Coreference Resolution
“Weir group” “whose” “headquarters” “U.S” “corporation” “power plant” “which” “Jiangsu” Weir Group Weir HQ United States Weir Group Weir Group Weir Plant Weir Plant Jiangsu .............. .......... .......... ... . ... ... . . ....... .......
SLIDE 17
Finite Mixture Model
SLIDE 18
Finite Mixture Model
Z1=
Weir Group
Z2=
Weir Group
Z3=
Weir HQ
SLIDE 19
Finite Mixture Model
P(Weir Group) = 0.2, ..... P(Weir HQ) = 0.5, Entity Distribution Z1=
Weir Group
Z2=
Weir Group
Z3=
Weir HQ
SLIDE 20
Finite Mixture Model
P(Weir Group) = 0.2, ..... P(Weir HQ) = 0.5, Entity Distribution Z1=
Weir Group
Z2=
Weir Group
Z3=
Weir HQ
W1=
“Weir Group”
W2=
“whose”
W3=
“headquart.”
SLIDE 21
Finite Mixture Model
P(Weir Group) = 0.2, ..... P(Weir HQ) = 0.5, Entity Distribution P(W | Weir Group): “Weir Group”=0.4, “whose”=0.2, ....... Mention Distribution Z1=
Weir Group
Z2=
Weir Group
Z3=
Weir HQ
W1=
“Weir Group”
W2=
“whose”
W3=
“headquart.”
SLIDE 22
Bayesian Finite Mixture Model
β
K Entity Distribution Z1=
Weir Group
W1=
“Weir Group”
Z2=
Weir Group
W2=
“whose”
Z3=
Weir HQ
W3=
“headquart.”
P(W | Weir Group): “Weir Group”=0.4, “whose”=0.2, ....... Mention Distribution
SLIDE 23
Bayesian Finite Mixture Model
β
K Entity Distribution Z1=
Weir Group
W1=
“Weir Group”
Z2=
Weir Group
W2=
“whose”
Z3=
Weir HQ
W3=
“headquart.”
P(W | Weir Group): “Weir Group”=0.4, “whose”=0.2, ....... Mention Distribution
This is how many entities there are
SLIDE 24
Bayesian Finite Mixture Model
β
K Entity Distribution Z1=
Weir Group
W1=
“Weir Group”
Z2=
Weir Group
W2=
“whose”
Z3=
Weir HQ
W3=
“headquart.”
P(W | Weir Group): “Weir Group”=0.4, “whose”=0.2, ....... Mention Distribution
SLIDE 25
Bayesian Finite Mixture Model
β
K K
φ
Z1=
Weir Group
W1=
“Weir Group”
Z2=
Weir Group
W2=
“whose”
Z3=
Weir HQ
W3=
“headquart.”
Mention Distribution Entity Distribution
SLIDE 26
Bayesian Finite Mixture Model
β
K K
φ
Z1=
Weir Group
W1=
“Weir Group”
Z2=
Weir Group
W2=
“whose”
Z3=
Weir HQ
W3=
“headquart.”
Mention Distribution Entity Distribution
How do you choose K?
SLIDE 27
Bayesian Finite Mixture Model
β
K K
φ
Z1=
Weir Group
W1=
“Weir Group”
Z2=
Weir Group
W2=
“whose”
Z3=
Weir HQ
W3=
“headquart.”
Mention Distribution Entity Distribution
SLIDE 28
Infinite Mixture Model
β
φ
Z1=
Weir Group
W1=
“Weir Group”
Z2=
Weir Group
W2=
“whose”
Z3=
Weir HQ
W3=
“headquart.”
Mention Distribution Entity Distribution
∞ ∞
SLIDE 29
Infinite Mixture Model
Drawn from a Dirichlet Process (DP) prior [Teh et al., 2006]
β
φ
Z1=
Weir Group
W1=
“Weir Group”
Z2=
Weir Group
W2=
“whose”
Z3=
Weir HQ
W3=
“headquart.”
Mention Distribution Entity Distribution
∞ ∞
SLIDE 30
Infinite Mixture Model
β
φ
Z1=
Weir Group
W1=
“Weir Group”
Z2=
Weir Group
W2=
“whose”
Z3=
Weir HQ
W3=
“headquart.”
Mention Distribution Entity Distribution
∞ ∞
SLIDE 31
Experimental Setup
SLIDE 32
Experimental Setup
- ACE 2004 English translations of
Arabic and Chinese Treebanks
SLIDE 33
Experimental Setup
- ACE 2004 English translations of
Arabic and Chinese Treebanks
- 95 Documents and 3,905 Mentions
SLIDE 34
Experimental Setup
- ACE 2004 English translations of
Arabic and Chinese Treebanks
- 95 Documents and 3,905 Mentions
- Given mention boundaries
SLIDE 35
Experimental Setup
- ACE 2004 English translations of
Arabic and Chinese Treebanks
- 95 Documents and 3,905 Mentions
- Given mention boundaries
- Evaluate on MUC F1 Measure
SLIDE 36
Infinite Mixture Model
MUC F1
50 60 70 80 90 100 Mixture
54.5
SLIDE 37
Infinite Mixture Model
MUC F1
The Weir Group , whose headquarters is in the U.S is a large specialized corporation. This power plant , which , will be situated in Jiangsu, has a large generation capacity.
50 60 70 80 90 100 Mixture
54.5
SLIDE 38
Infinite Mixture Model
MUC F1
The Weir Group , whose headquarters is in the U.S is a large specialized corporation. This power plant , which , will be situated in Jiangsu, has a large generation capacity.
50 60 70 80 90 100 Mixture
54.5
SLIDE 39
Infinite Mixture Model
MUC F1
The Weir Group , whose headquarters is in the U.S is a large specialized corporation. This power plant , which , will be situated in Jiangsu, has a large generation capacity.
50 60 70 80 90 100 Mixture
54.5
SLIDE 40
Infinite Mixture Model
MUC F1
The Weir Group , whose headquarters is in the U.S is a large specialized corporation. This power plant , which , will be situated in Jiangsu, has a large generation capacity.
50 60 70 80 90 100 Mixture
54.5 Pronouns lumped into
their own clusters!
SLIDE 41
Enriching Mention Model
W Z
SLIDE 42
Enriching Mention Model
W Z
Non-Pronoun
SLIDE 43
Enriching Mention Model
Pronoun
W Z W Z
Non-Pronoun
SLIDE 44
Enriching Mention Model
Pronoun
W Z N W Z
Non-Pronoun
SLIDE 45
Enriching Mention Model
Pronoun
W Z N
Number Sing, Plural
W Z
Non-Pronoun
SLIDE 46
Enriching Mention Model
Pronoun
W Z N W Z
Non-Pronoun
SLIDE 47
Enriching Mention Model
Pronoun
W Z G N W Z
Non-Pronoun
SLIDE 48
Enriching Mention Model
Pronoun
W Z G N
Gender M,F,N
W Z
Non-Pronoun
SLIDE 49
Enriching Mention Model
Pronoun
W Z G N W Z
Non-Pronoun
SLIDE 50
Enriching Mention Model
Pronoun
W Z T G N W Z
Non-Pronoun
SLIDE 51
Enriching Mention Model
Pronoun
W Z T G N W Z
Non-Pronoun
EntityType PERS, LOC, ORG, MISC
SLIDE 52
Enriching Mention Model
Pronoun
W Z T G N W Z
Non-Pronoun
SLIDE 53
Enriching Mention Model
Pronoun
W Z T G N
SLIDE 54
Enriching Mention Model
Entity Parameters
φ ∞ Pronoun
W Z T G N
SLIDE 55
Enriching Mention Model
W | SING, MALE, PERS “he”:0.5, “him”: 0.3,... W | PL, NEUT, ORG “they”:0.3, “it”: 0.2,... Entity Parameters
φ ∞
Pronoun Parameters
Pronoun
W Z T G N
SLIDE 56
Enriching Mention Model
Entity Parameters
φ ∞
Pronoun Parameters ψ
Pronoun
W Z T G N
SLIDE 57
Enriching Mention Model
W Z
Non-Pronoun Pronoun
W Z T G N
SLIDE 58
Enriching Mention Model
W Z G N T Pronoun Non-pronoun
SLIDE 59
Enriching Mention Model
M W Z G N T Pronoun Non-pronoun
SLIDE 60
Enriching Mention Model
M W Z G N T Pronoun Non-pronoun
Mention Type Proper, Pronoun, Nominal
SLIDE 61
Enriching Mention Model
M W Z G N T Pronoun Non-pronoun
SLIDE 62
Enriching Mention Model
β ∞
φ ∞
W W Z Z
..... .....
SLIDE 63
Enriching Mention Model
β ∞
φ ∞
Z W Z W G N T G N T M M
..... .....
SLIDE 64
Pronoun Head Model
50 60 70 80 90 100 Mixture Pronoun
64.1 54.5
MUC F1
SLIDE 65
Pronoun Head Model
50 60 70 80 90 100 Mixture Pronoun
64.1 54.5
MUC F1
The Weir Group , whose headquarters is in the U.S is a large specialized corporation. This power plant , which , will be situated in Jiangsu, has a large generation capacity.
SLIDE 66
Pronoun Head Model
50 60 70 80 90 100 Mixture Pronoun
64.1 54.5
MUC F1
The Weir Group , whose headquarters is in the U.S is a large specialized corporation. This power plant , which , will be situated in Jiangsu, has a large generation capacity.
SLIDE 67
Pronoun Head Model
50 60 70 80 90 100 Mixture Pronoun
64.1 54.5
MUC F1
The Weir Group , whose headquarters is in the U.S is a large specialized corporation. This power plant , which , will be situated in Jiangsu, has a large generation capacity.
Should be coreferent with recent “power plant” entity.
SLIDE 68
Pronoun Head Model
50 60 70 80 90 100 Mixture Pronoun
64.1 54.5
MUC F1
The Weir Group , whose headquarters is in the U.S is a large specialized corporation. This power plant , which , will be situated in Jiangsu, has a large generation capacity.
SLIDE 69
Salience Model
SLIDE 70
Salience Model
L
SLIDE 71
Salience Model
Entity Activation 1 1.0 2 0.0 L
SLIDE 72
Salience Model
Entity Activation 1 1.0 2 0.0 Z L
SLIDE 73
Salience Model
Entity Activation 1 1.0 2 0.0 Z L S
SLIDE 74
Salience Model
Entity Activation 1 1.0 2 0.0 Salience Values TOP, HIGH, MED, LOW, NONE Z L S
SLIDE 75
Salience Model
Entity Activation 1 1.0 2 0.0 Salience Values TOP, HIGH, MED, LOW, NONE Z L M S
SLIDE 76
Salience Model
Entity Activation 1 1.0 2 0.0 Salience Values TOP, HIGH, MED, LOW, NONE Z L M S Mention Type Proper, Pronoun, Nominal
SLIDE 77
Salience Model
Z1 M1 L1 S1
SLIDE 78
Salience Model
Z1 M1 L1 S1 Entity Activation 1 0.0 2 0.0
SLIDE 79
Salience Model
Z1 M1 L1 S1 Entity Activation 1 0.0 2 0.0 Ent 1
SLIDE 80
Salience Model
Z1 M1 L1 S1 Entity Activation 1 0.0 2 0.0 Ent 1 NONE
SLIDE 81
Salience Model
Z1 M1 L1 S1 Entity Activation 1 0.0 2 0.0 Ent 1 NONE PROPER
SLIDE 82
Salience Model
Z1 M1 L1 S1 Z2 M2 L2 S2 Entity Activation 1 0.0 2 0.0
SLIDE 83
Salience Model
Z1 M1 L1 S1 Entity Activation 1 1.0 2 0.0 Z2 M2 L2 S2 Entity Activation 1 0.0 2 0.0
SLIDE 84
Salience Model
Z1 M1 L1 S1 Entity Activation 1 1.0 2 0.0 Ent 2 Z2 M2 L2 S2 Entity Activation 1 0.0 2 0.0
SLIDE 85
Salience Model
Z1 M1 L1 S1 NONE Entity Activation 1 1.0 2 0.0 Ent 2 Z2 M2 L2 S2 Entity Activation 1 0.0 2 0.0
SLIDE 86
Salience Model
Z1 M1 L1 S1 NONE Entity Activation 1 1.0 2 0.0 PROPER Ent 2 Z2 M2 L2 S2 Entity Activation 1 0.0 2 0.0
SLIDE 87
Salience Model
Z1 M1 L1 S1 Z2 M2 L2 S2 Z3 M3 L3 S3 Entity Activation 1 1.0 2 0.0 Entity Activation 1 0.0 2 0.0
SLIDE 88
Salience Model
Entity Activation 1 0.5 2 1.0 Z1 M1 L1 S1 Z2 M2 L2 S2 Z3 M3 L3 S3 Entity Activation 1 1.0 2 0.0 Entity Activation 1 0.0 2 0.0
SLIDE 89
Salience Model
Entity Activation 1 0.5 2 1.0 Ent 2 Z1 M1 L1 S1 Z2 M2 L2 S2 Z3 M3 L3 S3 Entity Activation 1 1.0 2 0.0 Entity Activation 1 0.0 2 0.0
SLIDE 90
Salience Model
TOP Entity Activation 1 0.5 2 1.0 Ent 2 Z1 M1 L1 S1 Z2 M2 L2 S2 Z3 M3 L3 S3 Entity Activation 1 1.0 2 0.0 Entity Activation 1 0.0 2 0.0
SLIDE 91
Salience Model
TOP Entity Activation 1 0.5 2 1.0 Ent 2 PRONOUN Z1 M1 L1 S1 Z2 M2 L2 S2 Z3 M3 L3 S3 Entity Activation 1 1.0 2 0.0 Entity Activation 1 0.0 2 0.0
SLIDE 92
Enriching Mention Model
β ∞
φ ∞
Z W Z W G N T G N T M M
..... .....
SLIDE 93
Enriching Mention Model
β ∞
φ ∞
Z W Z W G N T G N T M M
..... .....
S L S L
SLIDE 94
Enriching Mention Model
β ∞
φ ∞
Z W Z W G N T G N T M M
..... .....
S L S L
SLIDE 95
Salience Model
50 60 70 80 90 100 Mixture Pronoun Salience
71.5 61.5 54.5
MUC F1
SLIDE 96
Salience Model
50 60 70 80 90 100 Mixture Pronoun Salience
71.5 61.5 54.5
MUC F1
The Weir Group , whose headquarters is in the U.S is a large specialized corporation. This power plant , which , will be situated in Jiangsu, has a large generation capacity.
SLIDE 97
Salience Model
TOP HIGH MID LOW NONE 1.000000000000000205
Proper Pronoun Nominal
SLIDE 98
Salience Model
TOP HIGH MID LOW NONE 1.000000000000000205
Proper Pronoun Nominal
SLIDE 99
Salience Model
TOP HIGH MID LOW NONE 1.000000000000000205
Proper Pronoun Nominal
SLIDE 100
Salience Model
TOP HIGH MID LOW NONE 1.000000000000000205
Proper Pronoun Nominal
SLIDE 101
Salience Model
TOP HIGH MID LOW NONE 1.000000000000000205
Proper Pronoun Nominal
SLIDE 102
Salience Model
TOP HIGH MID LOW NONE 1.000000000000000205
Proper Pronoun Nominal
SLIDE 103
Salience Model
TOP HIGH MID LOW NONE 1.000000000000000205
Proper Pronoun Nominal
SLIDE 104
Global Coreference Resolution
SLIDE 105
Global Coreference Resolution
SLIDE 106
Global Coreference Resolution
SLIDE 107
Global Coreference Resolution
SLIDE 108
Global Coreference Resolution
Global Entities
SLIDE 109
HDP Model
β ∞
φ ∞
Z W Z W G N T G N T M M
..... .....
S L S L
SLIDE 110
HDP Model
ψ N
HDP Model
β ∞
φ ∞
Z W Z W G N T G N T M M
.. ..
S L S L
SLIDE 111
HDP Model
∞
β0
ψ ψ N
β ∞
Z W Z W G N T G N T M M
.. ..
S L S L φ ∞
SLIDE 112
HDP Model
∞
β0
ψ
Global Entity Distribution drawn from a DP
ψ N
β ∞
Z W Z W G N T G N T M M
.. ..
S L S L φ ∞
SLIDE 113
HDP Model
∞
β0
ψ ψ N
β ∞
Z W Z W G N T G N T M M
.. ..
S L S L φ ∞
SLIDE 114
HDP Model
∞
β0
ψ ψ N
β ∞
Z W Z W G N T G N T M M
.. ..
S L S L φ ∞
Document Entity Distribution subsampled from Global Distr.
SLIDE 115
HDP Model
∞
β0
ψ ψ N
β ∞
Z W Z W G N T G N T M M
.. ..
S L S L φ ∞
SLIDE 116
HDP Model
50 60 70 80 90 100 Mixture Pronoun Salience HDP
72.5 71.5 64.1 54.5
MUC F1
SLIDE 117
HDP Model
50 60 70 80 90 100 Mixture Pronoun Salience HDP
72.5 71.5 64.1 54.5
MUC F1
The Weir Group , whose headquarters is in the U.S is a large specialized corporation. This power plant , which , will be situated in Jiangsu, has a large generation capacity.
SLIDE 118
HDP Model
SLIDE 119
HDP Model
Bush he Rice
SLIDE 120
HDP Model
Bush he Rice Rice Bush she
SLIDE 121
MUC6 Formal Experiments
Dataset # Docs. P R F MUC6 60 80.8 52.8 63.9 +DRYRUN 251 79.1 59.7 68.0 +NWIRE 381 80.4 62.4 70.3
- MUC6: 30 train / test documents
- Our Unsupervised Result
- Recent Supervised Result
- 73.4 F1 [McCallum and Wellner, 2004]
SLIDE 122
ACE Formal Experiments
- ACE 2004 English NWIRE
- 64.2 F1 [This paper] Unsupervised
- 67.5 F1 [Denis et al., 2007] Supervised
SLIDE 123
Summary
- Fully generative unsupervised
Bayesian nonparemetric coref model
- Sequential model of local attentional
state at the document level
- HDP global coreference model
- Broadly competitive with many
supervised results
SLIDE 124
Thanks! Questions?
SLIDE 125
Enriching Mention Model
W Z
SLIDE 126
Enriching Mention Model
W Z Entity Parameters “group”: 0.3, “its”: 0.2, ... Word | Entity
SLIDE 127
Enriching Mention Model
W Z “group”: 0.3, “its”: 0.2, ... Entity Parameters Word | Entity
SLIDE 128
Enriching Mention Model
W Z “group”: 0.3, “its”: 0.2, ... Entity Parameters Word | Entity Pronoun Non-pronoun
SLIDE 129
Enriching Mention Model
W Z “group”: 0.3, “its”: 0.2, ... Entity Parameters Word | Entity Pronoun Non-pronoun
SLIDE 130
Enriching Mention Model
W Z “group”: 0.3, “its”: 0.2, ... Entity Parameters T Word | Entity Pronoun Non-pronoun
SLIDE 131
Enriching Mention Model
W Z “group”: 0.3, “its”: 0.2, ... Entity Parameters T Word | Entity ORG: 0.5, LOC: 0.2, ... Entity Type | Entity Pronoun Non-pronoun
SLIDE 132
Enriching Mention Model
W Z “group”: 0.3, “its”: 0.2, ... Entity Parameters N T Word | Entity ORG: 0.5, LOC: 0.2, ... Entity Type | Entity Pronoun Non-pronoun
SLIDE 133
Enriching Mention Model
W Z “group”: 0.3, “its”: 0.2, ... Entity Parameters N T Word | Entity ORG: 0.5, LOC: 0.2, ... Entity Type | Entity PL: 0.6, SING: 0.4 Number | Entity Pronoun Non-pronoun
SLIDE 134
Enriching Mention Model
W Z “group”: 0.3, “its”: 0.2, ... Entity Parameters G N T Word | Entity ORG: 0.5, LOC: 0.2, ... Entity Type | Entity PL: 0.6, SING: 0.4 Number | Entity Pronoun Non-pronoun
SLIDE 135
Enriching Mention Model
W Z “group”: 0.3, “its”: 0.2, ... Entity Parameters G N T Word | Entity ORG: 0.5, LOC: 0.2, ... Entity Type | Entity PL: 0.6, SING: 0.4 Number | Entity
NEUT: 0.7, M: 0.2, F: 0.1
Gender | Entity Pronoun Non-pronoun
SLIDE 136
Enriching Mention Model
W Z Entity Parameters G N T Pronoun Non-pronoun
φ ∞
SLIDE 137
Enriching Mention Model
W Z G N T Pronoun Non-pronoun Entity Parameters
φ ∞
Pronoun Parameters
SLIDE 138
Enriching Mention Model
W | PERS, SING, MALE “he”:0.5, “him”: 0.5 W | ORG, PL, NEUT “they”:0.3, “it”: 0.2,... W Z G N T Pronoun Non-pronoun Entity Parameters
φ ∞
Pronoun Parameters
SLIDE 139
Enriching Mention Model
W Z G N T Pronoun Non-pronoun Entity Parameters
φ ∞
Pronoun Parameters
SLIDE 140
Enriching Mention Model
W Z G N T Pronoun Non-pronoun Entity Parameters
φ ∞
Pronoun Parameters ψ
SLIDE 141
Enriching Mention Model
W Z G N T Pronoun Parameters ψ Pronoun Non-pronoun Entity Parameters
φ ∞
SLIDE 142
Enriching Mention Model
M W Z G N T Pronoun Parameters ψ Pronoun Non-pronoun Entity Parameters
φ ∞
SLIDE 143