Gaussian Mixture Latent Vector Grammars Yanpeng Zhao Liwen Zhang - - PowerPoint PPT Presentation

School of Information Science and Technology ShanghaiTech University ACL 2018

Yanpeng Zhao Liwen Zhang Kewei Tu

Gaussian Mixture Latent Vector Grammars

PCFG Parsing & Limitations

S VP NP P me V found NP P He

He found me

Grammars Prob.

S → NP VP 1.0 VP → V NP 0.2 NP → DT NP 0.5 NP → NP NP 0.3 … … P → He 1.0 P → me 1.0 V → found 1.0 … …

PCFGs

Constituency Parsing

2

PCFG Parsing & Limitations

S VP NP P me V found NP P He

He found me

T1 =

S VP NP P me V found NP P He

Grammars Prob.

S → NP VP 1.0 VP → V NP 0.2 NP → DT NP 0.5 NP → NP NP 0.3 … … P → He 1.0 P → me 1.0 V → found 1.0 … …

PCFGs

Constituency Parsing

3

PCFG Parsing & Limitations

S VP NP P me V found NP P He

He found me

T1 = T2 =

S VP NP P He V found NP P me S VP NP P me V found NP P He

Grammars Prob.

S → NP VP 1.0 VP → V NP 0.2 NP → DT NP 0.5 NP → NP NP 0.3 … … P → He 1.0 P → me 1.0 V → found 1.0 … …

PCFGs

Constituency Parsing

4

PCFG Parsing & Limitations

S VP NP P me V found NP P He

He found me

T1 = T2 =

P(T1) = P(T2)

He found me 6= me found He

S VP NP P He V found NP P me S VP NP P me V found NP P He

Grammars Prob.

S → NP VP 1.0 VP → V NP 0.2 NP → DT NP 0.5 NP → NP NP 0.3 … … P → He 1.0 P → me 1.0 V → found 1.0 … …

PCFGs

limitations Constituency Parsing

5

Tree Annotation & Lexicalization

SˆROOT VPˆS-BD-Z NPˆVP-O P-O me V-BD-Z found NPˆS P-Z He S-found VP-found NP-me P-me me V-found found NP-He P-He He Tree Annotation Lexicalization

[Johnson. 1998; Klein et al. 2003] [Collins. 1997; Charniak. 2000]

6

Latent Variable Grammars (LVGs)

S[x1] VP[x4] NP[x6] P[x7] me V[x5] found NP[x2] P[x3] He

Annotated parse tree

S 0 VP 0 NP 2 P 3 me V 1 found NP 2 P 0 He

A subtype parse tree

• Discrete latent variables
• Model a finite number of nonterminal subtypes

x1, x2, x3 . . .

[Matsuzaki et al. 2005; Petrov et al. 2007]

7

Refining Syntactic Category

SˆROOT VPˆS-BD-Z NPˆVP-O P-O me V-BD-Z found NPˆS P-Z He S-found VP-found NP-me P-me me V-found found NP-He P-He He S[x1] VP[x4] NP[x6] P[x7] me V[x5] found NP[x2] P[x3] He

Tree annotation Lexicalization LVGs

[Klein et al. 2003] [Charniak. 2000] [Petrov et al. 2007] F1: 85.7 F1: 89.6 F1: 90.1

8

Latent Vector Grammars (LVeGs)

• Continuous latent vectors
• Model an infinite number of nonterminal subtypes

x1, x2, x3 . . .

S[x1] VP[x4] NP[x6] P[x7] me V[x5] found NP[x2] P[x3] He

S [0.05, 0.2] VP [3.4, 0.9] NP [1.3, 0.7] P [0.5, 1.4] me V [0.1, 0.2] found NP [0.4, 1.7] P [0.3, 2.1] He

Annotated parse tree A subtype parse tree

9

LVGs vs LVeGs

S[x1] VP[x4] NP[x6] P[x7] me V[x5] found NP[x2] P[x3] He S[x1] VP[x4] NP[x6] P[x7] me V[x5] found NP[x2] P[x3] He

(Latent Variable Grammars) (Latent Vector Grammars)

10

LVGs vs LVeGs

S[x1] VP[x4] NP[x6] P[x7] me V[x5] found NP[x2] P[x3] He S[x1] VP[x4] NP[x6] P[x7] me V[x5] found NP[x2] P[x3] He

Discrete latent variables Annotated rule NP[x2] P[x3]

x1, x2, x3 . . .

Continuous latent vectors Annotated rule NP[x2] P[x3]

x1, x2, x3 . . .

(Latent Variable Grammars) (Latent Vector Grammars)

11

LVGs vs LVeGs

S[x1] VP[x4] NP[x6] P[x7] me V[x5] found NP[x2] P[x3] He S[x1] VP[x4] NP[x6] P[x7] me V[x5] found NP[x2] P[x3] He

Discrete latent variables Annotated rule A finite set of subtype rules Subtype rule

NP[x2] P[x3]

x1, x2, x3 . . .

NP 0 P 1

Continuous latent vectors Annotated rule An infinite set of subtype rules Subtype rule

NP[x2] P[x3]

x1, x2, x3 . . .

NP [2.3, 1.7] P [0.4, 1.3]

(Latent Variable Grammars) (Latent Vector Grammars)

12

LVGs vs LVeGs

S[x1] VP[x4] NP[x6] P[x7] me V[x5] found NP[x2] P[x3] He S[x1] VP[x4] NP[x6] P[x7] me V[x5] found NP[x2] P[x3] He

Discrete latent variables Annotated rule A finite set of subtype rules Subtype rule

• Prob. of the subtype rule:

NP[x2] P[x3]

x1, x2, x3 . . .

NP 0 P 1

WNPP(x2 = 0, x3 = 1)

Continuous latent vectors Annotated rule An infinite set of subtype rules Subtype rule Weight density of the subtype rule:

NP[x2] P[x3]

x1, x2, x3 . . .

NP [2.3, 1.7] P [0.4, 1.3] WNPP(x2 = [2.3, 1.7], x3 = [0.4, 1.3])

(Latent Variable Grammars) (Latent Vector Grammars)

13

S[x1] VP[x4] NP[x6] P[x7] me V[x5] found NP[x2] P[x3] He

LVGs as Special Case of LVeGs

Discrete Variables One-hot Vectors

14

S[x1] VP[x4] NP[x6] P[x7] me V[x5] found NP[x2] P[x3] He

LVGs as Special Case of LVeGs

Wr(a, b) = X

x,y

ΘABba × δ(a − ax) × δ(b − by)

Dirac Delta Function

Discrete Variables One-hot Vectors

When represented by LVeGs, the rule weight function of 
 is defined as

r : A B

15

CVGs as Special Case of LVeGs

In CVGs, a rule is scored by

s(p(1)) = exp((vB,C)T p(1)) × p(P (1) BC)

PCFGs Parent Vector

(P (2), p(2) = x5 = f ⇣ W (A,P (1))h a

p(1)

i⌘ ) (P (1), p(1) = x4 = f ⇣ W (B,C)hb

c

i⌘ ) (C, c = x3) (B, b = x2) (A, a = x1)

[Socher et al., 2011 & 2013] (Compositional Vector Grammars)

16

CVGs as Special Case of LVeGs

In CVGs, a rule is scored by

s(p(1)) = exp((vB,C)T p(1)) × p(P (1) BC)

PCFGs Parent Vector

When represented by LVeGs, the rule weight function of 
 is defined as

Wr(a, b, c) = s(p) × δ(a − p)

(P (2), p(2) = x5 = f ⇣ W (A,P (1))h a

p(1)

i⌘ ) (P (1), p(1) = x4 = f ⇣ W (B,C)hb

c

i⌘ ) (C, c = x3) (B, b = x2) (A, a = x1)

CVGs LVeGs

r : A BC [Socher et al., 2011 & 2013] (Compositional Vector Grammars)

17

Parsing with LV(e)Gs

q(A BC, i, k, j) = s(A BC, i, k, j) sI(S, 1, n)

T ∗

q = argmax T ∈G(w)

Y

e∈T

q(e)

Max-Rule-Prod

Need to compute the posteriors of anchored rules:

S 0 VP 0 NP 2 P 3 me V 1 found NP 2 P 0 He S 1 VP 4 NP 3 P 0 me V 2 found NP 0 P 1 He S 0 VP 4 NP 2 P 3 me V 2 found NP 2 P 3 He

…

S VP NP P me V found NP P He

subtype parse trees unannotated parse tree what we have what we want [Petrov et al., 2007] < r, i, k, j >

18

Gaussian Mixture LVeGs (GM-LVeGs)

Formulate rule weight functions with Gaussian mixtures

• for a rule , its weight function is defined as

r : A BC

Wr(r) =

Kr

X

i=1

ρr,iN(r|µr,i, Σr,i) , where r = [a; b; c], and a, b, c ∈ Rd .

19

Gaussian Mixture LVeGs (GM-LVeGs)

Therefore, the Inside Score Function can be computed efficiently (similarly for the outside score function and expectations)

sA

I (a, i, j) =

X

ABC∈R k=i,··· ,j−1

ZZ WABC(a, b, c)×sB

I (b, i, k)

×sC

I (c, k + 1, j) dbdc

Formulate rule weight functions with Gaussian mixtures

• for a rule , its weight function is defined as

r : A BC

Advantage: Gaussian mixtures are closed under integral, product, and summation

Wr(r) =

Kr

X

i=1

ρr,iN(r|µr,i, Σr,i) , where r = [a; b; c], and a, b, c ∈ Rd .

20

Pruning Techniques

Inside Score Function with LVeGs

sA

I (a, i, j) =

X

ABC∈R k=i,··· ,j−1

ZZ WABC(a, b, c)×sB

I (b, i, k)

×sC

I (c, k + 1, j) dbdc

21

Pruning Techniques

Inside Score Function with LVeGs Component Pruning

• Limit the number of Gaussian components of inside and
• utside score functions

sA

I (a, i, j) =

X

ABC∈R k=i,··· ,j−1

ZZ WABC(a, b, c)×sB

I (b, i, k)

×sC

I (c, k + 1, j) dbdc

22

Pruning Techniques

Component Pruning

• Limit the number of Gaussian components of inside and
• utside score functions

Constituent Pruning

• Prune less probable syntactic categories for every span
• Consider only the constituents in k-best parses

sA

I (a, i, j) =

X

ABC∈R k=i,··· ,j−1

ZZ WABC(a, b, c)×sB

I (b, i, k)

×sC

I (c, k + 1, j) dbdc

Inside Score Function with LVeGs

23

Learning with GM-LVeGs

∂L(Θ) ∂Θr =

m

X

i=1

Z ✓∂Wr(r) ∂Θr × EP (t|wi)[fr(t)] − EP (t|Ti)[fr(t)] Wr(r) ◆ dr

L(Θ) = − log

m

Y

i=1

P(Ti|wi; Θ)

Discriminative learning Optimizing using Adam

Closed form solution exists when Gaussians are diagonal.
 Therefore, learning becomes possible.

[Kingma et al., 2014]

24

Experiments

Compared Models

• LVG-G: Generative training method (parsing & tagging)
• LVG-D: Discriminative training method (parsing & tagging)
• GM-LVeG-D: Diagonal Gaussians (tagging)
• GM-LVeG-S: Spherical Gaussians (parsing & tagging)

S VP PP NP NN telescope DT the IN with VP NP NNS stars VBD saw NP PRP He

PRP VBD NNS IN DT NN He saw stars with the telescope

HMMs: a special case of PCFGs Constituency Parsing Part-of-speech (POS) tagging

25

Results on Tagging

Universal Dependencies 1.4 (UD): English, French, German, Russian, Spanish, Indonesian, Finnish, and Italian treebanks. T/S: token/sentence accuracy.

Model WSJ English French German Russian T S T S T S T S T S LVG-D-16 96.62 48.74 92.31 52.67 93.75 34.90 87.38 20.98 81.91 12.25 LVG-G-16 96.78 50.88 93.30 57.54 94.52 34.90 88.92 24.05 84.03 16.63 GM-LVeG-D 96.99 53.10 93.66 59.46 94.73 39.60 89.11 24.77 84.21 17.84 GM-LVeG-S 97.00 53.11 93.55 58.11 94.74 39.26 89.14 25.58 84.06 18.44 Model Spanish Indonesian Finnish Italian T S T S T S T S LVG-D-16 92.47 24.82 89.27 20.29 83.81 19.29 94.81 45.19 LVG-G-16 93.21 27.37 90.09 21.19 85.01 20.53 95.46 48.26 GM-LVeG-D 93.76 32.48 90.24 21.72 85.27 23.30 95.61 50.72 GM-LVeG-S 93.52 30.66 90.12 21.72 85.35 22.07 95.62 49.69

(Discriminative) (Generative) (Diagonal) (Spherical)

26

Results on Parsing

• LVG-G/D (Petrov and Klein 2008a)
• Multi-Scale Grammars (Petrov and Klein 2008b)
• Berkeley Parser (Petrov and Klein 2007)
• CVG (SU-RNN) (Socher et al., 2013)

(exact match) Model dev (all) test ≤ 40 test (all) F1 F1 EX F1 EX LVG-G-16 88.70 35.80 LVG-D-16 89.30 39.40 Multi-Scale 89.70 39.60 89.20 37.20 Berkeley Parser 90.60 39.10 90.10 37.10 CVG (SU-RNN) 91.20 91.10 90.40 GM-LVeG-S 91.24 91.38 41.51 91.02 39.24

27

Summary

We propose Latent Vector Grammars (LVeGs)

• Each nonterminal associated with a continuous latent vector

space representing the set of nonterminal subtypes

• Latent variable grammars (LVGs) and compositional vector

grammars (CVGs) as special cases of LVeGs

We propose Gaussian Mixture LVeGs (GM-LVeGs)

• Rule weight functions formulated with Gaussian mixtures
• Efficient estimation in learning and inference
• Competitive results on POS tagging and constituency parsing

28

Future Work

• Automatically determine the component number of each

rule weight function (split, merge, regularization)

• Incorporate contextual information of words and

constituents (neural networks, non-diagonal Gaussians)

• Have a single continuous space for subtypes of all the

nonterminals (similarity between nonterminals)

• Study representation power of LVeGs with different kinds
• f Gaussians (spherical/diagonal/non-diagonal), and

compare to LVGs, CVGs, etc.

29