Sparsity in Dependency Grammar Induction Jennifer Gillenwater 1 - - PowerPoint PPT Presentation

Sparsity in Dependency Grammar Induction

Jennifer Gillenwater1 Kuzman Ganchev1 Jo˜ ao Gra¸ ca2 Ben Taskar1 Fernando Pereira3

1Computer & Information Science

University of Pennsylvania

2L2F INESC-ID, Lisboa, Portugal 3Google, Inc.

July 12, 2010

1/9

Outline

A generative dependency parsing model

2/9

Outline

A generative dependency parsing model The ambiguity problem this model faces

2/9

Outline

A generative dependency parsing model The ambiguity problem this model faces Previous attempts to reduce ambiguity

2/9

Outline

A generative dependency parsing model The ambiguity problem this model faces Previous attempts to reduce ambiguity How posteriors provide a good measure of ambiguity

2/9

Outline

A generative dependency parsing model The ambiguity problem this model faces Previous attempts to reduce ambiguity How posteriors provide a good measure of ambiguity Applying posterior regularization to the likelihood objective

2/9

Outline

A generative dependency parsing model The ambiguity problem this model faces Previous attempts to reduce ambiguity How posteriors provide a good measure of ambiguity Applying posterior regularization to the likelihood objective Success with respect to EM and parameter prior baselines

2/9

Dependency model with valence

(Klein and Manning, ACL 2004)

x y Regularization N creates V sparse ADJ grammars N pθ(x, y) = θroot(V )

3/9

Dependency model with valence

(Klein and Manning, ACL 2004)

x y Regularization N creates V sparse ADJ grammars N pθ(x, y) = θroot(V ) ·θstop(nostop|V ,right,false) · θchild(N|V ,right)

3/9

Dependency model with valence

(Klein and Manning, ACL 2004)

x y Regularization N creates V sparse ADJ grammars N pθ(x, y) = θroot(V ) ·θstop(nostop|V ,right,false) · θchild(N|V ,right) ·θstop(stop|V ,right,true) · θstop(nostop|V ,left,false) · θchild(N|V ,left) . . .

3/9

Traditional objective optimization

Traditional objective: marginal log likelihood max

θ

L(θ) = EX[log pθ(x)] = EX[log

• y

pθ(x, y)]

4/9

Traditional objective optimization

Traditional objective: marginal log likelihood max

θ

L(θ) = EX[log pθ(x)] = EX[log

• y

pθ(x, y)] Optimization method: expectation maximization (EM)

4/9

Traditional objective optimization

Traditional objective: marginal log likelihood max

θ

L(θ) = EX[log pθ(x)] = EX[log

• y

pθ(x, y)] Optimization method: expectation maximization (EM) Problem: EM may learn a very ambiguous grammar

Too many non-zero probabilities Ex: V → N should have non-zero probability, but V → DET, V → JJ, V → PRP$, etc. should be 0

4/9

Previous approaches to improving performance

Structural annealing1

1

Smith and Eisner, ACL 2006

2

Headden et al., NAACL 2009

3

Liang et al., EMNLP 2007; Johnson et al., NIPS 2007; Cohen et al., NIPS 2008, NAACL 2009 5/9

Previous approaches to improving performance

Structural annealing1 L(θ′): Model extension2

1

Smith and Eisner, ACL 2006

2

Headden et al., NAACL 2009

3

Liang et al., EMNLP 2007; Johnson et al., NIPS 2007; Cohen et al., NIPS 2008, NAACL 2009 5/9

Previous approaches to improving performance

Structural annealing1 L(θ′): Model extension2 L(θ) + log p(θ): Parameter regularization3

Tend to reduce unique # of children per parent, rather than directly reducing # of unique parent-child pairs θchild(Y |X,dir) = posterior(X→Y )

1

Smith and Eisner, ACL 2006

2

Headden et al., NAACL 2009

3

Liang et al., EMNLP 2007; Johnson et al., NIPS 2007; Cohen et al., NIPS 2008, NAACL 2009 5/9

Ambiguity measure using posteriors: L1/∞

Intuition: True # of unique parent tags for a child tag is small

N → N V → N ADJ → N N → V V → V ADJ → V N → ADJ V → ADJ ADJ → ADJ

6/9

Ambiguity measure using posteriors: L1/∞

N → N V → N ADJ → N N → V V → V ADJ → V N → ADJ V → ADJ ADJ → ADJ

Sparsity N is V working V

1

6/9

Ambiguity measure using posteriors: L1/∞

N → N V → N ADJ → N N → V V → V ADJ → V N → ADJ V → ADJ ADJ → ADJ

Sparsity N is V working V

1

Sparsity N is V working V

1

6/9

Ambiguity measure using posteriors: L1/∞

N → N V → N ADJ → N N → V V → V ADJ → V N → ADJ V → ADJ ADJ → ADJ

Sparsity N is V working V

1

Sparsity N is V working V

1

Use V good ADJ grammars N

1

6/9

Ambiguity measure using posteriors: L1/∞

N → N V → N ADJ → N N → V V → V ADJ → V N → ADJ V → ADJ ADJ → ADJ

Sparsity N is V working V

1

Sparsity N is V working V

1

Use V good ADJ grammars N

1

Use V good ADJ grammars N

1

6/9

Ambiguity measure using posteriors: L1/∞

N → N V → N ADJ → N N → V V → V ADJ → V N → ADJ V → ADJ ADJ → ADJ

Sparsity N is V working V

1

Sparsity N is V working V

1

Use V good ADJ grammars N

1

Use V good ADJ grammars N

1 max ↓ sum = 3 ← 1 1 1

6/9

Measuring ambiguity on distributions over trees

For a distribution pθ(y | x) instead of gold trees:

N → N V → N ADJ → N N → V V → V ADJ → V N → ADJ V → ADJ ADJ → ADJ

7/9

Measuring ambiguity on distributions over trees

N → N V → N ADJ → N N → V V → V ADJ → V N → ADJ V → ADJ ADJ → ADJ

Sparsity N is V working V 0.4 0.6

1

7/9

Measuring ambiguity on distributions over trees

N → N V → N ADJ → N N → V V → V ADJ → V N → ADJ V → ADJ ADJ → ADJ

Sparsity N is V working V 0.4 0.6

1

Sparsity N is V working V 0.4 0.6

.4 .6

7/9

Measuring ambiguity on distributions over trees

N → N V → N ADJ → N N → V V → V ADJ → V N → ADJ V → ADJ ADJ → ADJ

Sparsity N is V working V 0.4 0.6

1

Sparsity N is V working V 0.4 0.6

.4 .6

Use V good ADJ grammars N 0.70.3

.7 .3

7/9

Measuring ambiguity on distributions over trees

N → N V → N ADJ → N N → V V → V ADJ → V N → ADJ V → ADJ ADJ → ADJ

Sparsity N is V working V 0.4 0.6

1

Sparsity N is V working V 0.4 0.6

.4 .6

Use V good ADJ grammars N 0.70.3

.7 .3

Use V good ADJ grammars N 0.4 0.6

.4 .6

7/9

Measuring ambiguity on distributions over trees

N → N V → N ADJ → N N → V V → V ADJ → V N → ADJ V → ADJ ADJ → ADJ

Sparsity N is V working V 0.4 0.6

1

Sparsity N is V working V 0.4 0.6

.4 .6

Use V good ADJ grammars N 0.70.3

.7 .3

Use V good ADJ grammars N 0.4 0.6

.4 .6 max ↓ sum = 3.3 ← 1 .3 .4 .6 .4 .6

7/9

Minimizing ambiguity through posterior regularization

E-Step qt(y | x) = arg min

q(y|x)

KL(q pθt)

8/9

Minimizing ambiguity through posterior regularization

E-Step qt(y | x) = arg min

q(y|x)

KL(q pθt) q(y | x) =

D N V N

♣ t ① t

q(root → xi) parent D N V N child D

♣ ✈ ♣ ✈

N

✉ ♣ ✉ r

V

♣ ♣ ♣ ♣

N

✉ r ✉ ♣

q(xi → xj) Probability

r s t

→

8/9

Minimizing ambiguity through posterior regularization

Apply E-step penalty L1/∞ on posteriors q(y | x) to induce sparsity

(Graca et al., NIPS 2007 & 2009)

E-Step qt(y | x) = arg min

q(y|x)

KL(q pθt) + σL1/∞(q(y | x)) q(y | x) =

D N V N

♣ t ① t

q(root → xi) parent D N V N child D

♣ ✈ ♣ ✈

N

♣ ♣ ② r

V

♣ ♣ ♣ ♣

N

♣ r ② ♣

q(xi → xj) Probability

r s t

→

8/9

Experimental results

English from Penn Treebank: state-of-the-art accuracy Learning Method Accuracy ≤ 10 ≤ 20 all PR (σ = 140) 62.1 53.8 49.1 LN families 59.3 45.1 39.0 SLN TieV & N 61.3 47.4 41.4 PR (σ = 140, λ = 1/3) 64.4 55.2 50.5 DD (α = 1, λ learned) 65.0 (±5.7)

9/9

Experimental results

English from Penn Treebank: state-of-the-art accuracy Learning Method Accuracy ≤ 10 ≤ 20 all PR (σ = 140) 62.1 53.8 49.1 LN families 59.3 45.1 39.0 SLN TieV & N 61.3 47.4 41.4 PR (σ = 140, λ = 1/3) 64.4 55.2 50.5 DD (α = 1, λ learned) 65.0 (±5.7) 11 other languages from CoNLL-X:

Dirichlet prior baseline: 1.5% average gain over EM Posterior regularization: 6.5% average gain over EM

9/9

Experimental results

English from Penn Treebank: state-of-the-art accuracy Learning Method Accuracy ≤ 10 ≤ 20 all PR (σ = 140) 62.1 53.8 49.1 LN families 59.3 45.1 39.0 SLN TieV & N 61.3 47.4 41.4 PR (σ = 140, λ = 1/3) 64.4 55.2 50.5 DD (α = 1, λ learned) 65.0 (±5.7) 11 other languages from CoNLL-X:

Dirichlet prior baseline: 1.5% average gain over EM Posterior regularization: 6.5% average gain over EM

Come see the poster for more details

9/9