Dynamic Feature Selection for Dependency Parsing He He, Hal Daum - - PowerPoint PPT Presentation
Dynamic Feature Selection for Dependency Parsing He He, Hal Daum III and Jason Eisner EMNLP 2013, Seattle Structured Prediction in NLP Part-of-Speech Tagging Parsing N N V Det N
2
Fruit flies like a banana . 果 蝇 喜欢 香蕉 。 N ⟶ N ⟶ V ⟶ Det ⟶ N ↓ ↓ ↓ ↓ ↓ Fruit flies like a banana
Machine Translation Parsing Part-of-Speech Tagging $ Fruit flies like a banana
3
Fruit flies like a banana . 果 蝇 喜欢 香蕉 。 N ⟶ N ⟶ V ⟶ Det ⟶ N ↓ ↓ ↓ ↓ ↓ Fruit flies like a banana
Machine Translation Parsing Part-of-Speech Tagging $ Fruit flies like a banana
4
⋮ ⋮ ⋮
Fruit flies like a banana
⋮ ⋮
N V D N V D N V D N V D N V D a banana like flies Fruit $
5
⋮ ⋮ ⋮
Fruit flies like a banana
⋮ ⋮
N V D N V D N V D N V D N V D a banana like flies Fruit $
token left token right token in-between token stem form bigram tag coarse tag length direction ...
6
⋮ ⋮ ⋮
Fruit flies like a banana
⋮ ⋮
N V D N V D N V D N V D N V D a banana like flies Fruit $
token left token right token in-between token stem form bigram tag coarse tag length direction ...
(head_token + mod_token) X (head_tag +mod_tag)
7
⋮ ⋮ ⋮
Fruit flies like a banana
⋮ ⋮
token left token right token in-between token stem form bigram tag coarse tag length direction ...
(head_token + mod_token) X (head_tag +mod_tag) N V D N V D N V D N V D N V D a banana like flies Fruit $
8
⋮ ⋮ ⋮
Fruit flies like a banana
⋮ ⋮
N V D N V D N V D N V D N V D a banana like flies Fruit $
token tag coarse tag length direction ...
9
⋮ ⋮ ⋮
Fruit flies like a banana
⋮ ⋮
a banana like flies Fruit $ N V D N V D N V D N V D N V D
token tag coarse tag length direction ...
10
⋮ ⋮ ⋮
Fruit flies like a banana
⋮ ⋮
a banana like flies Fruit $ N V D N V D N V D N V D N V D
token tag coarse tag length direction ...
11
⋮ ⋮ ⋮
Fruit flies like a banana
⋮ ⋮
a banana like flies Fruit $ N V D N V D N V D N V D N V D
token tag coarse tag length direction ...
12
Bulgarian Chinese English German Japanese Portuguese Swedish
1 2 3 4 5 6 Ours Baseline S p e e d u p
13
.
This time
,
the firms were ready
$
14
.
This time
,
the firms were ready
$
length: 1 direction: right modifier_token: were head_token: firms head_tag: noun
15
.
This time
,
the firms were ready
$
the
$
This time
,
firms were ready
.
16
17
?
18
Average Sentence
10 20 30 40 50 60
19
This the firms ready
20
This the firms ready
+0.63
21
This the firms ready
+0.1 +0.63 +0.7
22
This the firms ready
+0.1 +0.63 +0.7 WINNER
23
This the firms ready
+0.1
+0.63 +0.7 WINNER
24
This the firms ready
+0.1
+0.63 +0.7 WINNER LOSER
25
This the firms ready
+0.1
+0.63 +0.7
WINNER LOSER
26
27
28
29
30
31
.
This time
,
the firms were ready $
32
.
This time
,
the firms were ready $
33
.
This time
,
the firms were ready $
34
.
This time
,
the firms were ready $
35
.
This time
,
the firms were ready $
36
.
This time
,
the firms were ready $
37
.
This time
,
the firms were ready $
38
.
This time
,
the firms were ready $
39
.
This time
,
the firms were ready $
40
.
This time
,
the firms were ready $
41
.
This time
,
the firms were ready $
42
.
This time
,
the firms were ready $
43
.
This time
,
the firms were ready $
44
.
This time
,
the firms were ready $
45
.
This time
,
the firms were ready $
46
.
This time
,
the firms were ready $
47
.
This time
,
the firms were ready $
48
.
This time
,
the firms were ready $
49
.
This time
,
the firms were ready $
50
.
This time
,
the firms were ready $
51
.
This time
,
the firms were ready $
52
.
This time
,
the firms were ready $
53
54
55
56
57
58
– Will definitely appear in the 1-best tree
– Have the same child as a winning edge – Form cycle with winning edges – Cross a winning edge (optional) – Share root ($) with a winning edge (optional)
– Add the next feature group to the remaining
59
60
61
62
63
64
65
head cPOS+ mod cPOS + in-between punct # 0.49 in-between cPOS 0.59 head POS + mod POS + in-between conj # 0.71 head POS + mod POS + in-between POS + dist 0.72 head token + mod cPOS + dist 0.80
66
head cPOS+ mod cPOS + in-between punct # 0.49 in-between cPOS 0.59 head POS + mod POS + in-between conj # 0.71 head POS + mod POS + in-between POS + dist 0.72 head token + mod cPOS + dist 0.80
67
68
69
70
71
72
– : …, 0.5, 0.8, 0.85
– Margins to the highest-scoring competing edge – Index of the next feature group
the firms
.
This time the firms were $
73
– : …, 0.5, 0.8, 0.85
– Margins to the highest-scoring competing edge – Index of the next feature group
the firms
.
This time the firms were $
74
– : …, 0.5, 0.8, 0.85
– Margins to the highest-scoring competing edge – Index of the next feature group
the firms
.
This time the firms were $
75
– Set of edges to classify at stage k – The dynamic features of those edges at stage k
76
– Set of edges to classify at stage k – The dynamic features of those edges at stage k
– Iterates between training and running a model – Learns to recover from past mistakes
77
– Gold edges always win – 96.47% UAS with 2.9% first-order features
.
This time
,
the firms were ready $
.
This time
,
the firms were ready $
78
79
– Penn Treebank: English – CoNLL-X: Bulgarian, Chinese, German, Japanese,
– MSTParser (McDonald et al., 2006)
– LibLinear (Fan et al., 2008)
80
81
82
Bulgarian Chinese English German Japanese Portuguese Swedish
1 2 3 4 5 6 DynFS Baseline S p e e d u p
83
Bulgarian Chinese English German Japanese Portuguese Swedish
99.3% 99.4% 99.5% 99.6% 99.7% 99.8% 99.9% 100.0% 100.1% 100.2% 100.3% DynFS Baseline R e l a t i v e a c c u r a c y
84
were ready
.
the
$
This time
,
firms were ready
.
85
Bulgarian Chinese English German Japanese Portuguese Swedish
1 2 3 4 5 6 7 8 9 DynFS Baseline S p e e d u p
86
Bulgarian Chinese English German Japanese Portuguese Swedish
99.3% 99.4% 99.5% 99.6% 99.7% 99.8% 99.9% 100.0% 100.1% 100.2% 100.3% DynFS Baseline R e l a t i v e a c c u r a c y
87
– Start with short edges (fully scored) – Add long edges in if needed
– Start with all edges (partially scored) – Quickly remove unneeded edges
88
Bulgarian Chinese English German Japanese Portuguese Swedish
1 2 3 4 5 6 DynFS VineP Baseline S p e e d u p
89
Bulgarian Chinese English German Japanese Portuguese Swedish
99.3% 99.4% 99.5% 99.6% 99.7% 99.8% 99.9% 100.0% 100.1% 100.2% 100.3% DynFS VineP Baseline R e l a t i v e a c c u r a c y
90
Bulgarian Chinese English German Japanese Portuguese Swedish
2 4 6 8 10 12 14 16 DynFS VineP Baseline S p e e d u p
91
Bulgarian Chinese English German Japanese Portuguese Swedish
99.3% 99.4% 99.5% 99.6% 99.7% 99.8% 99.9% 100.0% 100.1% 100.2% 100.3% DynFS VineP Baseline R e l a t i v e a c c u r a c y
92
93
94
VB CD: → 18 VB CD: ← 3 NN VBG: → 22 NN VBG: ← 11 ... .
This time
,
the firms were ready $
95
– State: all the information helping us to make
– Action: things we choose to do – Reward: criteria for evaluating actions – Policy: the “brain” that makes the decision
– Maximize the expected future reward
96
– reward = accuracy + λ∙speed
– Delayed reward – Long time to converge
– Mimic the oracle – Reduced to supervised classification problem
the firms
97
– (near) optimal performance – generate target action in any given state
the firms time
,
the
98
– Different distribution at training and test time
99
Bulgarian Chinese English German Japanese Portuguese Swedish
0.00% 5.00% 10.00% 15.00% 20.00% 25.00% 30.00% 35.00% 40.00% 45.00% DynFS F e a t u r e c
t
100
Bulgarian Chinese English German Japanese Portuguese Swedish
0.00% 10.00% 20.00% 30.00% 40.00% 50.00% 60.00% 70.00% 80.00% DynFS F e a t u r e c
t
101
.
102
.