Algorithms for NLP CS 11-711 Fall 2020 Lecture 14: Graph-based - - PowerPoint PPT Presentation

Emma Strubell

Algorithms for NLP

CS 11-711 · Fall 2020

Lecture 14: Graph-based dependency parsing

Announcements

2

■ No recitation on Friday (Tartan Community Day).

Dependency parsing

3

Dependency Relations wn w1 w2 s2 ... s1 sn

Parser

Input buffer Stack

Oracle

■ Transition-based (shift-reduce) parsing:

Dependency parsing

3

Dependency Relations wn w1 w2 s2 ... s1 sn

Parser

Input buffer Stack

Oracle

■ Transition-based (shift-reduce) parsing: ■ Greedy choice of local transitions guided

by a good classifier.

Dependency parsing

3

Dependency Relations wn w1 w2 s2 ... s1 sn

Parser

Input buffer Stack

Oracle

■ Transition-based (shift-reduce) parsing: ■ Greedy choice of local transitions guided

by a good classifier.

■ Examples: MaltParser [Nivre et al. 2008],

Stack LSTM [Dyer et al. 2015]

Dependency parsing

3

Dependency Relations wn w1 w2 s2 ... s1 sn

Parser

Input buffer Stack

Oracle

■ Transition-based (shift-reduce) parsing: ■ Greedy choice of local transitions guided

by a good classifier.

■ Examples: MaltParser [Nivre et al. 2008],

Stack LSTM [Dyer et al. 2015]

■ Graph-based dependency parsing:

Dependency parsing

3

root Book that flight 12 4 4 5 6 8 7 5 7

Dependency Relations wn w1 w2 s2 ... s1 sn

Parser

Input buffer Stack

Oracle

■ Transition-based (shift-reduce) parsing: ■ Greedy choice of local transitions guided

by a good classifier.

■ Examples: MaltParser [Nivre et al. 2008],

Stack LSTM [Dyer et al. 2015]

■ Graph-based dependency parsing: ■ Given scores for every pair of words, find

the (globally) highest scoring set of edges.

Dependency parsing

3

root Book that flight 12 4 4 5 6 8 7 5 7

Dependency Relations wn w1 w2 s2 ... s1 sn

Parser

Input buffer Stack

Oracle

■ Transition-based (shift-reduce) parsing: ■ Greedy choice of local transitions guided

by a good classifier.

■ Examples: MaltParser [Nivre et al. 2008],

Stack LSTM [Dyer et al. 2015]

■ Graph-based dependency parsing: ■ Given scores for every pair of words, find

the (globally) highest scoring set of edges.

■ Examples: MSTParser [McDonald et al.

2005], TurboParser [Martins et al. 2009], Deep Biaffine [Dozat et al. 2017]

Dependency parsing

3

root Book that flight 12 4 4 5 6 8 7 5 7

root Book that flight 12 4 4 5 6 8 7 5 7

Graph-based dependency parsing

4

root Book that flight 12 4 4 5 6 8 7 5 7

■ Edge-factored (or arc-factored) approaches:

Graph-based dependency parsing

4

root Book that flight 12 4 4 5 6 8 7 5 7

■ Edge-factored (or arc-factored) approaches: ■ Score of a tree decomposes as sum of edge scores:

Graph-based dependency parsing

4

Ψ(y, w; θ) = X

i

r

− →j∈y ψ(i

r

− → j, w, θ)

root Book that flight 12 4 4 5 6 8 7 5 7

■ Edge-factored (or arc-factored) approaches: ■ Score of a tree decomposes as sum of edge scores: ■ Start with a fully-connected directed graph

Graph-based dependency parsing

4

Ψ(y, w; θ) = X

i

r

− →j∈y ψ(i

r

− → j, w, θ)

root Book that flight 12 4 4 5 6 8 7 5 7

■ Edge-factored (or arc-factored) approaches: ■ Score of a tree decomposes as sum of edge scores: ■ Start with a fully-connected directed graph ■ How to infer the highest scoring tree?

Graph-based dependency parsing

4

Ψ(y, w; θ) = X

i

r

− →j∈y ψ(i

r

− → j, w, θ)

root Book that flight 12 4 4 5 6 8 7 5 7

■ Edge-factored (or arc-factored) approaches: ■ Score of a tree decomposes as sum of edge scores: ■ Start with a fully-connected directed graph ■ How to infer the highest scoring tree? ■ Find a maximum directed spanning tree:

Chu and Liu (1965) and Edmonds (1967) algorithm

Graph-based dependency parsing

4

Ψ(y, w; θ) = X

i

r

− →j∈y ψ(i

r

− → j, w, θ)

Chu-Liu-Edmonds algorithm

5

function MAXSPANNINGTREE(G=(V,E), root,score) returns spanning tree F←[] T’←[] score’←[] for each v ∈ V do bestInEdge←argmaxe=(u,v)∈ E score[e] F←F ∪ bestInEdge for each e=(u,v) ∈ E do score’[e]←score[e] − score[bestInEdge] if T=(V,F) is a spanning tree then return it else C←a cycle in F G’←CONTRACT(G,C) T’←MAXSPANNINGTREE(G’,root,score’) T←EXPAND(T’, C) return T function CONTRACT(G,C) returns contracted graph function EXPAND(T, C) returns expanded graph

Chu-Liu-Edmonds algorithm

5

function MAXSPANNINGTREE(G=(V,E), root,score) returns spanning tree F←[] T’←[] score’←[] for each v ∈ V do bestInEdge←argmaxe=(u,v)∈ E score[e] F←F ∪ bestInEdge for each e=(u,v) ∈ E do score’[e]←score[e] − score[bestInEdge] if T=(V,F) is a spanning tree then return it else C←a cycle in F G’←CONTRACT(G,C) T’←MAXSPANNINGTREE(G’,root,score’) T←EXPAND(T’, C) return T function CONTRACT(G,C) returns contracted graph function EXPAND(T, C) returns expanded graph

select best incoming edge for each node

Chu-Liu-Edmonds algorithm

5

function MAXSPANNINGTREE(G=(V,E), root,score) returns spanning tree F←[] T’←[] score’←[] for each v ∈ V do bestInEdge←argmaxe=(u,v)∈ E score[e] F←F ∪ bestInEdge for each e=(u,v) ∈ E do score’[e]←score[e] − score[bestInEdge] if T=(V,F) is a spanning tree then return it else C←a cycle in F G’←CONTRACT(G,C) T’←MAXSPANNINGTREE(G’,root,score’) T←EXPAND(T’, C) return T function CONTRACT(G,C) returns contracted graph function EXPAND(T, C) returns expanded graph

select best incoming edge for each node subtract its score from all incoming edges

Chu-Liu-Edmonds algorithm

5

function MAXSPANNINGTREE(G=(V,E), root,score) returns spanning tree F←[] T’←[] score’←[] for each v ∈ V do bestInEdge←argmaxe=(u,v)∈ E score[e] F←F ∪ bestInEdge for each e=(u,v) ∈ E do score’[e]←score[e] − score[bestInEdge] if T=(V,F) is a spanning tree then return it else C←a cycle in F G’←CONTRACT(G,C) T’←MAXSPANNINGTREE(G’,root,score’) T←EXPAND(T’, C) return T function CONTRACT(G,C) returns contracted graph function EXPAND(T, C) returns expanded graph

select best incoming edge for each node subtract its score from all incoming edges stopping condition

Chu-Liu-Edmonds algorithm

5

function MAXSPANNINGTREE(G=(V,E), root,score) returns spanning tree F←[] T’←[] score’←[] for each v ∈ V do bestInEdge←argmaxe=(u,v)∈ E score[e] F←F ∪ bestInEdge for each e=(u,v) ∈ E do score’[e]←score[e] − score[bestInEdge] if T=(V,F) is a spanning tree then return it else C←a cycle in F G’←CONTRACT(G,C) T’←MAXSPANNINGTREE(G’,root,score’) T←EXPAND(T’, C) return T function CONTRACT(G,C) returns contracted graph function EXPAND(T, C) returns expanded graph

select best incoming edge for each node subtract its score from all incoming edges stopping condition contract nodes if there are cycles

Chu-Liu-Edmonds algorithm

5

function MAXSPANNINGTREE(G=(V,E), root,score) returns spanning tree F←[] T’←[] score’←[] for each v ∈ V do bestInEdge←argmaxe=(u,v)∈ E score[e] F←F ∪ bestInEdge for each e=(u,v) ∈ E do score’[e]←score[e] − score[bestInEdge] if T=(V,F) is a spanning tree then return it else C←a cycle in F G’←CONTRACT(G,C) T’←MAXSPANNINGTREE(G’,root,score’) T←EXPAND(T’, C) return T function CONTRACT(G,C) returns contracted graph function EXPAND(T, C) returns expanded graph

select best incoming edge for each node subtract its score from all incoming edges stopping condition contract nodes if there are cycles recursively compute MST

Chu-Liu-Edmonds algorithm

5

function MAXSPANNINGTREE(G=(V,E), root,score) returns spanning tree F←[] T’←[] score’←[] for each v ∈ V do bestInEdge←argmaxe=(u,v)∈ E score[e] F←F ∪ bestInEdge for each e=(u,v) ∈ E do score’[e]←score[e] − score[bestInEdge] if T=(V,F) is a spanning tree then return it else C←a cycle in F G’←CONTRACT(G,C) T’←MAXSPANNINGTREE(G’,root,score’) T←EXPAND(T’, C) return T function CONTRACT(G,C) returns contracted graph function EXPAND(T, C) returns expanded graph

select best incoming edge for each node subtract its score from all incoming edges stopping condition contract nodes if there are cycles recursively compute MST expand contracted nodes

Chu-Liu-Edmonds algorithm

6

■ Select best incoming edge for each node

root Book 12 that 7 flight 8 12 4 4 5 6 8 7 5 7

Chu-Liu-Edmonds algorithm

7

■ Subtract its score from all incoming edges

root Book 12 that 7 flight 8 12 4 4 5 6 8 7 5 7

• 12
• 12
• 12
• 7
• 7
• 7
• 8
• 8
• 8

root Book 12 that 7 flight 8

• 4
• 3
• 2
• 6
• 1
• 7

Chu-Liu-Edmonds algorithm

8

■ Subtract its score from all incoming edges

root Book 12 that 7 flight 8

• 4
• 3
• 2
• 6
• 1
• 7

Chu-Liu-Edmonds algorithm

9

■ Contract nodes if there are cycles

■ Contract nodes if there are cycles

Chu-Liu-Edmonds algorithm

10

root Book tf

• 3
• 4
• 7
• 1
• 6
• 2

■ Recursively compute MST

Chu-Liu-Edmonds algorithm

11

root Book tf

• 1
• 3
• 4
• 7
• 1
• 6
• 2

■ Expand contracted nodes

Chu-Liu-Edmonds algorithm

12

root Book that flight Deleted from cycle

Chu-Liu-Edmonds algorithm

13

function MAXSPANNINGTREE(G=(V,E), root,score) returns spanning tree F←[] T’←[] score’←[] for each v ∈ V do bestInEdge←argmaxe=(u,v)∈ E score[e] F←F ∪ bestInEdge for each e=(u,v) ∈ E do score’[e]←score[e] − score[bestInEdge] if T=(V,F) is a spanning tree then return it else C←a cycle in F G’←CONTRACT(G,C) T’←MAXSPANNINGTREE(G’,root,score’) T←EXPAND(T’, C) return T function CONTRACT(G,C) returns contracted graph function EXPAND(T, C) returns expanded graph

runtime?

Chu-Liu-Edmonds algorithm

13

function MAXSPANNINGTREE(G=(V,E), root,score) returns spanning tree F←[] T’←[] score’←[] for each v ∈ V do bestInEdge←argmaxe=(u,v)∈ E score[e] F←F ∪ bestInEdge for each e=(u,v) ∈ E do score’[e]←score[e] − score[bestInEdge] if T=(V,F) is a spanning tree then return it else C←a cycle in F G’←CONTRACT(G,C) T’←MAXSPANNINGTREE(G’,root,score’) T←EXPAND(T’, C) return T function CONTRACT(G,C) returns contracted graph function EXPAND(T, C) returns expanded graph

runtime? naive: O(n3)

Chu-Liu-Edmonds algorithm

13

function MAXSPANNINGTREE(G=(V,E), root,score) returns spanning tree F←[] T’←[] score’←[] for each v ∈ V do bestInEdge←argmaxe=(u,v)∈ E score[e] F←F ∪ bestInEdge for each e=(u,v) ∈ E do score’[e]←score[e] − score[bestInEdge] if T=(V,F) is a spanning tree then return it else C←a cycle in F G’←CONTRACT(G,C) T’←MAXSPANNINGTREE(G’,root,score’) T←EXPAND(T’, C) return T function CONTRACT(G,C) returns contracted graph function EXPAND(T, C) returns expanded graph

runtime? naive: O(n3) fancy: O(n2 + nlogn)

Chu-Liu-Edmonds algorithm

13

function MAXSPANNINGTREE(G=(V,E), root,score) returns spanning tree F←[] T’←[] score’←[] for each v ∈ V do bestInEdge←argmaxe=(u,v)∈ E score[e] F←F ∪ bestInEdge for each e=(u,v) ∈ E do score’[e]←score[e] − score[bestInEdge] if T=(V,F) is a spanning tree then return it else C←a cycle in F G’←CONTRACT(G,C) T’←MAXSPANNINGTREE(G’,root,score’) T←EXPAND(T’, C) return T function CONTRACT(G,C) returns contracted graph function EXPAND(T, C) returns expanded graph

runtime? naive: O(n3) fancy: O(n2 + nlogn) what about labeled parsing?

root Book that flight 12 4 4 5 6 8 7 5 7

■ Edge-factored (or arc-factored) approaches: ■ Score of a tree decomposes as sum of edge scores: ■ Start with a fully-connected directed graph ■ How to infer the highest scoring tree? ■ Find a maximum directed spanning tree:

Chu and Liu (1965) and Edmonds (1967) algorithm

Graph-based dependency parsing

14

Ψ(y, w; θ) = X

i

r

− →j∈y ψ(i

r

− → j, w, θ)

root Book that flight 12 4 4 5 6 8 7 5 7

■ Edge-factored (or arc-factored) approaches: ■ Score of a tree decomposes as sum of edge scores: ■ Start with a fully-connected directed graph ■ How to infer the highest scoring tree? ■ Find a maximum directed spanning tree:

Chu and Liu (1965) and Edmonds (1967) algorithm

Graph-based dependency parsing

14

Ψ(y, w; θ) = X

i

r

− →j∈y ψ(i

r

− → j, w, θ)

■ For projective trees: Eisner’s algorithm [Eisner 1996]

Graph-based dependency parsing

15

Ψ(y, w; θ) = X

i

r

− →j∈y ψ(i

r

− → j, w, θ)

■ Edge-factored (or arc-factored) approaches: score of a tree decomposes

as sum of edge scores.

■ Can also define higher-order models: score decomposes as a sum of scores

• f local subgraphs.

Graph-based dependency parsing

15

Ψ(y, w; θ) = X

i

r

− →j∈y ψ(i

r

− → j, w, θ)

First order h m Second order h s m g h m Third order g h s m h t s m

■ Edge-factored (or arc-factored) approaches: score of a tree decomposes

as sum of edge scores.

■ Can also define higher-order models: score decomposes as a sum of scores

• f local subgraphs.

■ Have efficient (polynomial time) algorithms for second [Eisner 1996] and

third order [Koo and Collins, 2010] projective dependency parsing.

Graph-based dependency parsing

15

Ψ(y, w; θ) = X

i

r

− →j∈y ψ(i

r

− → j, w, θ)

First order h m Second order h s m g h m Third order g h s m h t s m

■ Edge-factored (or arc-factored) approaches: score of a tree decomposes

as sum of edge scores.

■ Can also define higher-order models: score decomposes as a sum of scores

• f local subgraphs.

■ Have efficient (polynomial time) algorithms for second [Eisner 1996] and

third order [Koo and Collins, 2010] projective dependency parsing.

■ Second order parsing is NP-hard for non-projective dependency graphs

[McDonald and Pereira, 2006]!

Graph-based dependency parsing

15

Ψ(y, w; θ) = X

i

r

− →j∈y ψ(i

r

− → j, w, θ)

First order h m Second order h s m g h m Third order g h s m h t s m

■ Edge-factored (or arc-factored) approaches: score of a tree decomposes

as sum of edge scores.

Graph-based dependency parsing

16

First order h m

Ψ(y, w; θ) = X

i

r

− →j∈y ψ(i

r

− → j, w, θ)

■ Edge-factored (or arc-factored) approaches: score of a tree decomposes

as sum of edge scores.

■ How to parameterize ?

Graph-based dependency parsing

16

ψ

First order h m

Ψ(y, w; θ) = X

i

r

− →j∈y ψ(i

r

− → j, w, θ)

■ Edge-factored (or arc-factored) approaches: score of a tree decomposes

as sum of edge scores.

Classic graph-based parsing features

17

■ Word forms, lemmas, parts of speech of the head word and its dependent ■ Corresponding features derived from the contexts before, after, between words ■ Word embeddings ■ Dependency relation ■ Direction of the relation (right or left) ■ Distance from the head to the dependent ■ Combinations of all of the above

Graph-based neural network parsers

18

back the bill will Janet

word embeddings per-token features

neural network

Graph-based neural network parsers

19

per-token features

neural network

back/VB the/DT bill/NN will/MD Janet/NNP

word embeddings pos embeddings

Graph-based neural network parsers

19

per-token features

neural network

edge features

…

root → Janet will → Janet back → Janet the → Janet bill → Janet Janet → bill

back/VB the/DT bill/NN will/MD Janet/NNP

word embeddings pos embeddings

Graph-based neural network parsers

19

per-token features

neural network

edge features

…

root → Janet will → Janet back → Janet the → Janet bill → Janet Janet → bill

back/VB the/DT bill/NN will/MD Janet/NNP

word embeddings pos embeddings

Graph-based neural network parsers

20

Graph-based neural network parsers

20

■ Concat + feed-forward [Kiperwasser and Goldberg, 2016]

LSTM f xthe concat LSTM f xbrown concat LSTM f xfox concat LSTM f xjumped concat LSTM f x∗ concat LSTM b s0 LSTM b s1 LSTM b s2 LSTM b s3 LSTM b s4 Vthe Vbrown Vfox Vjumped V∗ MLP MLP MLP MLP

+

Graph-based neural network parsers

20

■ Concat + feed-forward [Kiperwasser and Goldberg, 2016]

LSTM f xthe concat LSTM f xbrown concat LSTM f xfox concat LSTM f xjumped concat LSTM f x∗ concat LSTM b s0 LSTM b s1 LSTM b s2 LSTM b s3 LSTM b s4 Vthe Vbrown Vfox Vjumped V∗ MLP MLP MLP MLP

+

stacked LSTM
 token encoder

Graph-based neural network parsers

20

■ Concat + feed-forward [Kiperwasser and Goldberg, 2016]

LSTM f xthe concat LSTM f xbrown concat LSTM f xfox concat LSTM f xjumped concat LSTM f x∗ concat LSTM b s0 LSTM b s1 LSTM b s2 LSTM b s3 LSTM b s4 Vthe Vbrown Vfox Vjumped V∗ MLP MLP MLP MLP

+

stacked LSTM
 token encoder concat head, dependent
 as input to MLP

Graph-based neural network parsers

20

■ Concat + feed-forward [Kiperwasser and Goldberg, 2016] ■ Hinge loss:

LSTM f xthe concat LSTM f xbrown concat LSTM f xfox concat LSTM f xjumped concat LSTM f x∗ concat LSTM b s0 LSTM b s1 LSTM b s2 LSTM b s3 LSTM b s4 Vthe Vbrown Vfox Vjumped V∗ MLP MLP MLP MLP

+

stacked LSTM
 token encoder concat head, dependent
 as input to MLP

max ⇣ 0, 1 max

y06=y

X

(h,m)2y0

MLP(vh vm) X ⌘ ⇣ X

2

+ X

(h,m)2y

MLP(vh vm) ⌘

Graph-based neural network parsers

20

■ Concat + feed-forward [Kiperwasser and Goldberg, 2016] ■ Hinge loss: ■ w/ loss augmented inference:

LSTM f xthe concat LSTM f xbrown concat LSTM f xfox concat LSTM f xjumped concat LSTM f x∗ concat LSTM b s0 LSTM b s1 LSTM b s2 LSTM b s3 LSTM b s4 Vthe Vbrown Vfox Vjumped V∗ MLP MLP MLP MLP

+

stacked LSTM
 token encoder concat head, dependent
 as input to MLP

max ⇣ 0, 1 max

y06=y

X

(h,m)2y0

MLP(vh vm) X ⌘ ⇣ X

2

+ X

(h,m)2y

MLP(vh vm) ⌘

max(0, 1 + score(x, y) X

• max

y06=y

X

part2y0

(scorelocal(x, part) +

part62y))

Graph-based neural network parsers

21

Graph-based neural network parsers

21

■ Biaffine classifier [Dozat and Manning, 2017]

... root ROOT Kim NNP

1 1 1 1

⊤ · · = BiLSTM: ri Embeddings: xi MLP: h(arc-dep)

i

, h(arc-head)

i

H(arc-dep) ⊕ 1 U (arc) H(arc-head) S(arc)

Graph-based neural network parsers

21

■ Biaffine classifier [Dozat and Manning, 2017]

... root ROOT Kim NNP

1 1 1 1

⊤ · · = BiLSTM: ri Embeddings: xi MLP: h(arc-dep)

i

, h(arc-head)

i

H(arc-dep) ⊕ 1 U (arc) H(arc-head) S(arc)

stacked LSTM
 token encoder

Graph-based neural network parsers

21

■ Biaffine classifier [Dozat and Manning, 2017]

... root ROOT Kim NNP

1 1 1 1

⊤ · · = BiLSTM: ri Embeddings: xi MLP: h(arc-dep)

i

, h(arc-head)

i

H(arc-dep) ⊕ 1 U (arc) H(arc-head) S(arc)

stacked LSTM
 token encoder MLPhead MLPdep

Graph-based neural network parsers

21

■ Biaffine classifier [Dozat and Manning, 2017]

... root ROOT Kim NNP

1 1 1 1

⊤ · · = BiLSTM: ri Embeddings: xi MLP: h(arc-dep)

i

, h(arc-head)

i

H(arc-dep) ⊕ 1 U (arc) H(arc-head) S(arc)

stacked LSTM
 token encoder MLPhead MLPdep biaffine classifier

si = Wri + b Fixed-class affine classifier s(arc)

i

=

• RU (1)

ri +

• Ru(2)

Variable-class biaffine classifier

Graph-based neural network parsers

21

■ Biaffine classifier [Dozat and Manning, 2017] ■ Locally normalized log loss.

... root ROOT Kim NNP

1 1 1 1

⊤ · · = BiLSTM: ri Embeddings: xi MLP: h(arc-dep)

i

, h(arc-head)

i

H(arc-dep) ⊕ 1 U (arc) H(arc-head) S(arc)

stacked LSTM
 token encoder MLPhead MLPdep biaffine classifier

si = Wri + b Fixed-class affine classifier s(arc)

i

=

• RU (1)

ri +

• Ru(2)

Variable-class biaffine classifier

■ How to parameterize ?

Graph-based dependency parsing

22

ψ

First order h m

Ψ(y, w; θ) = X

i

r

− →j∈y ψ(i

r

− → j, w, θ)

■ Edge-factored (or arc-factored) approaches: score of a tree decomposes

as sum of edge scores.

■ How to parameterize ?

Graph-based dependency parsing

22

ψ

First order h m

Ψ(y, w; θ) = X

i

r

− →j∈y ψ(i

r

− → j, w, θ)

■ Edge-factored (or arc-factored) approaches: score of a tree decomposes

as sum of edge scores.

■ How to learn θ ?

■ How to parameterize ?

Graph-based dependency parsing

22

ψ

First order h m

Ψ(y, w; θ) = X

i

r

− →j∈y ψ(i

r

− → j, w, θ)

■ Edge-factored (or arc-factored) approaches: score of a tree decomposes

as sum of edge scores.

■ How to learn θ ? ■ Locally normalized: predict each head and its label, softmax, log loss.

■ How to parameterize ?

Graph-based dependency parsing

22

ψ

First order h m

Ψ(y, w; θ) = X

i

r

− →j∈y ψ(i

r

− → j, w, θ)

■ Edge-factored (or arc-factored) approaches: score of a tree decomposes

as sum of edge scores.

■ How to learn θ ? ■ Locally normalized: predict each head and its label, softmax, log loss. ■ Hinge loss: Find the best scoring tree, and penalize edges not in the gold

tree (with a margin).

■ How to parameterize ?

Graph-based dependency parsing

22

ψ

First order h m

Ψ(y, w; θ) = X

i

r

− →j∈y ψ(i

r

− → j, w, θ)

■ Edge-factored (or arc-factored) approaches: score of a tree decomposes

as sum of edge scores.

■ How to learn θ ? ■ Locally normalized: predict each head and its label, softmax, log loss. ■ Hinge loss: Find the best scoring tree, and penalize edges not in the gold

tree (with a margin).

■ Globally normalized CRF: can compute marginals/partition function using a

variant of Kirchhoff’s Matrix-Tree Theorem [Tutte, 1984; Koo et al. 2007].

Graph-based vs. transition-based parsing?

23

Graph-based vs. transition-based parsing?

23

■ Transition-based ■ Fast ■ Greedy / local inference ■ Maybe closer to humans?

Dependency Relations wn w1 w2 s2 ... s1 sn

Parser

Input buffer Stack

Oracle

Graph-based vs. transition-based parsing?

23

■ Transition-based ■ Fast ■ Greedy / local inference ■ Maybe closer to humans? ■ Graph-based ■ Slow ■ Exact inference ■ More accurate

Dependency Relations wn w1 w2 s2 ... s1 sn

Parser

Input buffer Stack

Oracle

root Book that flight 12 4 4 5 6 8 7 5 7

Graph-based vs. transition-based parsing?

24

Dependency parsing: accuracy vs. speed Accuracy (UAS) Speed (sentences/sec)

Graph-based vs. transition-based parsing?

24

[Zhang & McDonald 2014] TurboParser [Martins et al. 2010]

Dependency parsing: accuracy vs. speed Accuracy (UAS) Speed (sentences/sec)

Graph-based

Graph-based vs. transition-based parsing?

24

MaltParser [Nivre et al. 2009] [Zhang & McDonald 2014] TurboParser [Martins et al. 2010]

Dependency parsing: accuracy vs. speed Accuracy (UAS) Speed (sentences/sec)

Graph-based Transition-based

Graph-based vs. transition-based parsing?

24

MaltParser [Nivre et al. 2009] [Zhang & McDonald 2014] TurboParser [Martins et al. 2010] [Rush & Petrov 2012;

Dependency parsing: accuracy vs. speed Accuracy (UAS) Speed (sentences/sec)

Graph-based Transition-based Structured prediction cascades Weiss & Taskar 2010]

Graph-based vs. transition-based parsing?

24

MaltParser [Nivre et al. 2009] [Chen & Manning 2014] [Zhang & McDonald 2014] TurboParser [Martins et al. 2010] [Rush & Petrov 2012;

Dependency parsing: accuracy vs. speed Accuracy (UAS) Speed (sentences/sec)

Graph-based Transition-based FF neural nets Structured prediction cascades Weiss & Taskar 2010]

Graph-based vs. transition-based parsing?

24

transition-based w/ dynamic feature selection MaltParser [Nivre et al. 2009] [Chen & Manning 2014] [Zhang & McDonald 2014] TurboParser [Martins et al. 2010] [Rush & Petrov 2012;

Dependency parsing: accuracy vs. speed Accuracy (UAS) Speed (sentences/sec)

Graph-based Transition-based FF neural nets Structured prediction cascades [Strubell et al. 2015] Weiss & Taskar 2010] [Strubell et al. 2015]

Graph-based vs. transition-based parsing?

25

transition-based w/ dynamic feature selection

Dependency parsing: accuracy vs. speed Accuracy (UAS) Speed (sentences/sec)

Graph-based Transition-based FF neural nets Structured prediction cascades

94 95 96

Graph-based vs. transition-based parsing?

25

transition-based w/ dynamic feature selection

Dependency parsing: accuracy vs. speed Accuracy (UAS) Speed (sentences/sec)

Graph-based Transition-based FF neural nets Structured prediction cascades

94 95 96

Stacked LSTMs [Dozat and Manning 2017]*

Graph-based vs. transition-based parsing?

25

transition-based w/ dynamic feature selection

Dependency parsing: accuracy vs. speed Accuracy (UAS) Speed (sentences/sec)

Graph-based Transition-based FF neural nets Structured prediction cascades

94 95 96

Stacked LSTMs [Dozat and Manning 2017]*

*on GPU!!

Graph-based vs. transition-based parsing?

25

transition-based w/ dynamic feature selection

Dependency parsing: accuracy vs. speed Accuracy (UAS) Speed (sentences/sec)

Graph-based Transition-based FF neural nets Structured prediction cascades

94 95 96

Stacked LSTMs [Dozat and Manning 2017]*

*on GPU!!

Announcements

26

■ No recitation on Friday (Tartan Community Day).