Algorithms for NLP CS 11-711 Fall 2020 Lecture 9: CRFs, neural - - PowerPoint PPT Presentation

Emma Strubell

Algorithms for NLP

CS 11-711 · Fall 2020

Lecture 9: CRFs, neural sequence labeling

Announcements

2

■ Project 2 released today after class: sequence labeling. ■ Due: October 16. ■ You will implement part-of-speech taggers for English and Norwegian: ■ HMM, BiLSTM, and BiLSTM-CRF. ■ Friday’s recitation will be an overview of P2.

Recap

3

Recap

3

■ HMMs: Natural extension of Naive Bayes to sequence labeling

Recap

3

■ HMMs: Natural extension of Naive Bayes to sequence labeling ■ Hard to add rich features of the input, e.g. affixes, capitalization, …

Recap

3

■ HMMs: Natural extension of Naive Bayes to sequence labeling ■ Hard to add rich features of the input, e.g. affixes, capitalization, … ■ Would like to train a discriminative model, like logistic regression, to directly model

the conditional probability of labels given inputs.

Recap

3

■ HMMs: Natural extension of Naive Bayes to sequence labeling ■ Hard to add rich features of the input, e.g. affixes, capitalization, … ■ Would like to train a discriminative model, like logistic regression, to directly model

the conditional probability of labels given inputs.

■ Logistic regression (MEMMs) suffer from the label bias problem.

Recap

3

■ HMMs: Natural extension of Naive Bayes to sequence labeling ■ Hard to add rich features of the input, e.g. affixes, capitalization, … ■ Would like to train a discriminative model, like logistic regression, to directly model

the conditional probability of labels given inputs.

■ Logistic regression (MEMMs) suffer from the label bias problem. ■ Solution: linear-chain CRFs.

Conditional random fields (CRFs)

4

■ Linear-chain CRFs: Globally-normalized discriminative sequence labeling models!

P(y | w) = exp(Ψ(w, y)) P

y0∈Y(w)

exp(Ψ(w, y0))

Conditional random fields (CRFs)

4

■ Linear-chain CRFs: Globally-normalized discriminative sequence labeling models!

P(y | w) = exp(Ψ(w, y)) P

y0∈Y(w)

exp(Ψ(w, y0))

Conditional random fields (CRFs)

4

y’ is an entire sequence

■ Linear-chain CRFs: Globally-normalized discriminative sequence labeling models!

P(y | w) = exp(Ψ(w, y)) P

y0∈Y(w)

exp(Ψ(w, y0))

Conditional random fields (CRFs)

4

y’ is an entire sequence

Ψ(w, y) =

M+1

X

m=1

ψ(w, ym, ym−1, m)

■ Linear-chain CRFs: Globally-normalized discriminative sequence labeling models!

P(y | w) = exp(Ψ(w, y)) P

y0∈Y(w)

exp(Ψ(w, y0))

Conditional random fields (CRFs)

4

y’ is an entire sequence decompose into local scores

Ψ(w, y) =

M+1

X

m=1

ψ(w, ym, ym−1, m)

■ Linear-chain CRFs: Globally-normalized discriminative sequence labeling models!

P(y | w) = exp(Ψ(w, y)) P

y0∈Y(w)

exp(Ψ(w, y0))

Conditional random fields (CRFs)

4

y’ is an entire sequence

y) =

M+1

X

m=1

θ · f(w, ym, ym−1, m)

decompose into local scores

Ψ(w, y) =

M+1

X

m=1

ψ(w, ym, ym−1, m)

■ Linear-chain CRFs: Globally-normalized discriminative sequence labeling models!

Conditional random fields (CRFs)

5

Ψ(w, y) =

M+1

X

m=1

θ · f(w, ym, ym−1, m)

■ Linear-chain CRFs: Globally-normalized discriminative sequence labeling models!

Conditional random fields (CRFs)

5

Ψ(w, y) =

M+1

X

m=1

θ · f(w, ym, ym−1, m)

Janet will back the bill

wm+1 wm wm-1 wm-2

<s>

w

■ Linear-chain CRFs: Globally-normalized discriminative sequence labeling models!

Conditional random fields (CRFs)

5

Ψ(w, y) =

M+1

X

m=1

θ · f(w, ym, ym−1, m)

Janet will back the bill

wm+1 wm wm-1 wm-2

VB?

<s>

w

■ Linear-chain CRFs: Globally-normalized discriminative sequence labeling models!

Conditional random fields (CRFs)

5

Ψ(w, y) =

M+1

X

m=1

θ · f(w, ym, ym−1, m)

1 … 1 … … 1 …

Janet will back the bill

wm+1 wm wm-1 wm-2

VB?

<s>

w

■ Linear-chain CRFs: Globally-normalized discriminative sequence labeling models!

Conditional random fields (CRFs)

5

Ψ(w, y) =

M+1

X

m=1

θ · f(w, ym, ym−1, m)

1 … 1 … … 1 …

Janet will back the bill

wm+1 wm wm-1 wm-2

NNP MD VB?

<s>

ym-1

f(w, VB, MD, 3)

w y

■ Linear-chain CRFs: Globally-normalized discriminative sequence labeling models!

Conditional random fields (CRFs)

5

Ψ(w, y) =

M+1

X

m=1

θ · f(w, ym, ym−1, m)

1 … 1 … … 1 …

Janet will back the bill

= f(w1, w2, w3, w4, VB, MD) wm-1 = will wm-1 = my wm = back wm+1 = ache ym-1 = MD

wm+1 wm wm-1 wm-2

NNP MD VB?

<s>

ym-1

f(w, VB, MD, 3)

w y

■ Linear-chain CRFs: Globally-normalized discriminative sequence labeling models!

Conditional random fields (CRFs)

5

Ψ(w, y) =

M+1

X

m=1

θ · f(w, ym, ym−1, m)

1 … 1 … … 1 …

Janet will back the bill

= f(w1, w2, w3, w4, VB, MD) wm-1 = will wm-1 = my wm = back wm+1 = ache ym-1 = MD

wm+1 wm wm-1 wm-2

NNP MD VB?

<s>

ym-1

K + F

f(w, VB, MD, 3)

w y

■ Linear-chain CRFs: Globally-normalized discriminative sequence labeling models!

Conditional random fields (CRFs)

5

Ψ(w, y) =

M+1

X

m=1

θ · f(w, ym, ym−1, m)

1 … 1 … … 1 …

Janet will back the bill

= f(w1, w2, w3, w4, VB, MD) wm-1 = will wm-1 = my wm = back wm+1 = ache ym-1 = MD

wm+1 wm wm-1 wm-2

NNP MD VB?

<s>

ym-1

K + F

f(w, VB, MD, 3)

w y

# labels

■ Linear-chain CRFs: Globally-normalized discriminative sequence labeling models!

Conditional random fields (CRFs)

5

Ψ(w, y) =

M+1

X

m=1

θ · f(w, ym, ym−1, m)

1 … 1 … … 1 …

Janet will back the bill

= f(w1, w2, w3, w4, VB, MD) wm-1 = will wm-1 = my wm = back wm+1 = ache ym-1 = MD

wm+1 wm wm-1 wm-2

NNP MD VB?

<s>

ym-1

K + F

f(w, VB, MD, 3)

w y

# features # labels

■ Linear-chain CRFs: Globally-normalized discriminative sequence labeling models!

Conditional random fields (CRFs)

5

Ψ(w, y) =

M+1

X

m=1

θ · f(w, ym, ym−1, m)

1 … 1 … … 1 …

Janet will back the bill

= f(w1, w2, w3, w4, VB, MD) wm-1 = will wm-1 = my wm = back wm+1 = ache ym-1 = MD

wm+1 wm wm-1 wm-2

NNP MD VB?

<s>

ym-1

K + F

(K − 1)(K + F)

f(w, VB, MD, 3)

w y

# features # labels

■ Linear-chain CRFs: Globally-normalized discriminative sequence labeling models!

Conditional random fields (CRFs)

6

■ Decoding: Direct application of Viterbi!

P(y | w) = exp(Ψ(w, y)) P

y0∈Y(w)

exp(Ψ(w, y0))

Conditional random fields (CRFs)

6

■ Decoding: Direct application of Viterbi!

P(y | w) = exp(Ψ(w, y)) P

y0∈Y(w)

exp(Ψ(w, y0))

Conditional random fields (CRFs)

6

■ Decoding: Direct application of Viterbi!

ˆ y = argmax

y

log P(y | w)

P(y | w) = exp(Ψ(w, y)) P

y0∈Y(w)

exp(Ψ(w, y0))

Conditional random fields (CRFs)

6

■ Decoding: Direct application of Viterbi!

ˆ y = argmax

y

log P(y | w)

= argmax

y

Ψ(y, w) − log X

y0∈Y(w)

exp Ψ(y0, w)

P(y | w) = exp(Ψ(w, y)) P

y0∈Y(w)

exp(Ψ(w, y0))

Conditional random fields (CRFs)

6

■ Decoding: Direct application of Viterbi!

ˆ y = argmax

y

log P(y | w)

= argmax

y

Ψ(y, w) − log X

y0∈Y(w)

exp Ψ(y0, w)

P(y | w) = exp(Ψ(w, y)) P

y0∈Y(w)

exp(Ψ(w, y0))

Conditional random fields (CRFs)

6

■ Decoding: Direct application of Viterbi!

ˆ y = argmax

y

log P(y | w)

= argmax

y

Ψ(y, w) − log X

y0∈Y(w)

exp Ψ(y0, w)

P(y | w) = exp(Ψ(w, y)) P

y0∈Y(w)

exp(Ψ(w, y0))

= argmax

y

Ψ(y, w)

same for all settings of y

Conditional random fields (CRFs)

6

■ Decoding: Direct application of Viterbi!

ˆ y = argmax

y

log P(y | w)

= argmax

y

Ψ(y, w) − log X

y0∈Y(w)

exp Ψ(y0, w)

P(y | w) = exp(Ψ(w, y)) P

y0∈Y(w)

exp(Ψ(w, y0))

= argmax

y

Ψ(y, w)

same for all settings of y

= argmax

y M+1

X

m=1

ψ(w, ym, ym−1, m)

Conditional random fields (CRFs)

6

■ Decoding: Direct application of Viterbi!

ˆ y = argmax

y

log P(y | w)

= argmax

y

Ψ(y, w) − log X

y0∈Y(w)

exp Ψ(y0, w)

P(y | w) = exp(Ψ(w, y)) P

y0∈Y(w)

exp(Ψ(w, y0))

= argmax

y

Ψ(y, w)

same for all settings of y

= argmax

y M+1

X

m=1

ψ(w, ym, ym−1, m)

= sm(ym, ym−1)

Conditional random fields (CRFs)

7

■ Decoding: Direct application of Viterbi!

= argmax

y M+1

X

m=1

ψ(w, ym, ym−1, m)

= sm(ym, ym−1)

ˆ y

Conditional random fields (CRFs)

7

■ Decoding: Direct application of Viterbi!

= argmax

y M+1

X

m=1

ψ(w, ym, ym−1, m)

= sm(ym, ym−1)

max

y1:M Ψ(w, y1:M) = max y1:M M+1

X

m=1

sm(ym, ym−1)

ˆ y

Conditional random fields (CRFs)

7

■ Decoding: Direct application of Viterbi!

= argmax

y M+1

X

m=1

ψ(w, ym, ym−1, m)

= sm(ym, ym−1)

max

y1:M Ψ(w, y1:M) = max y1:M M+1

X

m=1

sm(ym, ym−1)

ˆ y

score of best tag sequence

• f length M

Conditional random fields (CRFs)

7

■ Decoding: Direct application of Viterbi!

= argmax

y M+1

X

m=1

ψ(w, ym, ym−1, m)

= sm(ym, ym−1)

max

y1:M Ψ(w, y1:M) = max y1:M M+1

X

m=1

sm(ym, ym−1)

ˆ y

score of best tag sequence

• f length M

= max

yM sM+1(h/si, yM) + max y1:M−1 M

X

m=1

sm(ym, ym−1)

Conditional random fields (CRFs)

7

■ Decoding: Direct application of Viterbi!

= argmax

y M+1

X

m=1

ψ(w, ym, ym−1, m)

= sm(ym, ym−1)

max

y1:M Ψ(w, y1:M) = max y1:M M+1

X

m=1

sm(ym, ym−1)

ˆ y

score of best tag sequence

• f length M

= max

yM sM+1(h/si, yM) + max y1:M−1 M

X

m=1

sm(ym, ym−1)

score of best tag sequence

• f length M-1

Conditional random fields (CRFs)

7

■ Decoding: Direct application of Viterbi!

= argmax

y M+1

X

m=1

ψ(w, ym, ym−1, m)

= sm(ym, ym−1)

max

y1:M Ψ(w, y1:M) = max y1:M M+1

X

m=1

sm(ym, ym−1)

ˆ y

score of best tag sequence

• f length M

= max

yM sM+1(h/si, yM) + max y1:M−1 M

X

m=1

sm(ym, ym−1)

score of best tag sequence

• f length M-1

score of most probable extension yM

Conditional random fields (CRFs)

7

■ Decoding: Direct application of Viterbi!

= argmax

y M+1

X

m=1

ψ(w, ym, ym−1, m)

= sm(ym, ym−1)

max

y1:M Ψ(w, y1:M) = max y1:M M+1

X

m=1

sm(ym, ym−1)

ˆ y

score of best tag sequence

• f length M

= max

yM sM+1(h/si, yM) + max y1:M−1 M

X

m=1

sm(ym, ym−1)

score of best tag sequence

• f length M-1

score of most probable extension yM same subproblem!

Conditional random fields (CRFs)

7

■ Decoding: Direct application of Viterbi!

= argmax

y M+1

X

m=1

ψ(w, ym, ym−1, m)

= sm(ym, ym−1)

max

y1:M Ψ(w, y1:M) = max y1:M M+1

X

m=1

sm(ym, ym−1)

ˆ y

score of best tag sequence

• f length M

= max

yM sM+1(h/si, yM) + max y1:M−1 M

X

m=1

sm(ym, ym−1)

score of best tag sequence

• f length M-1

score of most probable extension yM same subproblem! ψ includes transition and emission/unary scores

Learning in CRFs

8

Learning in CRFs

8

■ As with logistic regression, weights θ are learned by minimizing negative log

likelihood:

` = −

N

X

i=1

log P(y(i) | w(i); ✓)

Learning in CRFs

8

■ As with logistic regression, weights θ are learned by minimizing negative log

likelihood:

` = −

N

X

i=1

log P(y(i) | w(i); ✓)

= −

N

X

i=1

θ · f(w(i), y(i)) + log X

y0∈Y(w(i))

exp ⇣ θ · f(w(i), y0) ⌘

Learning in CRFs

8

■ As with logistic regression, weights θ are learned by minimizing negative log

likelihood:

` = −

N

X

i=1

log P(y(i) | w(i); ✓)

= −

N

X

i=1

θ · f(w(i), y(i)) + log X

y0∈Y(w(i))

exp ⇣ θ · f(w(i), y0) ⌘

sum over all possible labelings

Learning in CRFs

8

■ As with logistic regression, weights θ are learned by minimizing negative log

likelihood:

■ Can be computed efficiently using forward algorithm.

` = −

N

X

i=1

log P(y(i) | w(i); ✓)

= −

N

X

i=1

θ · f(w(i), y(i)) + log X

y0∈Y(w(i))

exp ⇣ θ · f(w(i), y0) ⌘

sum over all possible labelings

Learning in CRFs

9

Learning in CRFs

9

■ Likelihood can be computed efficiently using forward algorithm.

Learning in CRFs

9

■ Likelihood can be computed efficiently using forward algorithm.

Define: αm(ym) = X

y1:m−1

exp

m

X

n=1

sn(yn, yn−1)

Learning in CRFs

9

■ Likelihood can be computed efficiently using forward algorithm.

Define: αm(ym) = X

y1:m−1

exp

m

X

n=1

sn(yn, yn−1)

= X

y1:m−1 m

Y

n=1

exp sn(yn, yn−1)

Learning in CRFs

9

■ Likelihood can be computed efficiently using forward algorithm. ■ As in decoding / Viterbi, can be decomposed into recursive substructure:

Define: αm(ym) = X

y1:m−1

exp

m

X

n=1

sn(yn, yn−1)

= X

y1:m−1 m

Y

n=1

exp sn(yn, yn−1)

= X

ym−1

(exp sm(ym, ym−1)) X

y1:m−2 m−1

Y

n=1

exp sn(yn, yn−1)

Learning in CRFs

9

■ Likelihood can be computed efficiently using forward algorithm. ■ As in decoding / Viterbi, can be decomposed into recursive substructure:

Define:

= X

ym−1

(exp sm(ym, ym−1)) × αm−1(ym−1)

αm(ym) = X

y1:m−1

exp

m

X

n=1

sn(yn, yn−1)

= X

y1:m−1 m

Y

n=1

exp sn(yn, yn−1)

= X

ym−1

(exp sm(ym, ym−1)) X

y1:m−2 m−1

Y

n=1

exp sn(yn, yn−1)

Learning in CRFs

9

■ Likelihood can be computed efficiently using forward algorithm. ■ As in decoding / Viterbi, can be decomposed into recursive substructure:

Define:

= X

ym−1

(exp sm(ym, ym−1)) × αm−1(ym−1)

sum instead of max

αm(ym) = X

y1:m−1

exp

m

X

n=1

sn(yn, yn−1)

= X

y1:m−1 m

Y

n=1

exp sn(yn, yn−1)

= X

ym−1

(exp sm(ym, ym−1)) X

y1:m−2 m−1

Y

n=1

exp sn(yn, yn−1)

Learning in CRFs

9

■ Likelihood can be computed efficiently using forward algorithm. ■ As in decoding / Viterbi, can be decomposed into recursive substructure:

Define:

= X

ym−1

(exp sm(ym, ym−1)) × αm−1(ym−1)

sum instead of max Viterbi is a special case of the max-product algorithm, forward is a special case of the sum-product algorithm.

αm(ym) = X

y1:m−1

exp

m

X

n=1

sn(yn, yn−1)

= X

y1:m−1 m

Y

n=1

exp sn(yn, yn−1)

= X

ym−1

(exp sm(ym, ym−1)) X

y1:m−2 m−1

Y

n=1

exp sn(yn, yn−1)

Learning in CRFs

9

■ Likelihood can be computed efficiently using forward algorithm. ■ As in decoding / Viterbi, can be decomposed into recursive substructure:

Define:

= X

ym−1

(exp sm(ym, ym−1)) × αm−1(ym−1)

sum instead of max Viterbi is a special case of the max-product algorithm, forward is a special case of the sum-product algorithm.

vm(k) = M

k02Y

sm(k, k0) ⊗ vm1(k0).

αm(ym) = X

y1:m−1

exp

m

X

n=1

sn(yn, yn−1)

= X

y1:m−1 m

Y

n=1

exp sn(yn, yn−1)

= X

ym−1

(exp sm(ym, ym−1)) X

y1:m−2 m−1

Y

n=1

exp sn(yn, yn−1)

Learning in CRFs

10

Learning in CRFs

10

■ As in logistic regression, gradient of the likelihood w.r.t. parameters is difference

between observed and expected feature counts: ` ✓j =

N

X

i=1

E[fj(w(i), y)] − fj(w(i), y(i))

Learning in CRFs

10

■ As in logistic regression, gradient of the likelihood w.r.t. parameters is difference

between observed and expected feature counts:

count of feature j for token sequence w(i), tag sequence y(i)

` ✓j =

N

X

i=1

E[fj(w(i), y)] − fj(w(i), y(i))

Learning in CRFs

10

■ As in logistic regression, gradient of the likelihood w.r.t. parameters is difference

between observed and expected feature counts:

■ Gradients can be computed by automatic differentiation!

count of feature j for token sequence w(i), tag sequence y(i)

` ✓j =

N

X

i=1

E[fj(w(i), y)] − fj(w(i), y(i))

Learning in CRFs

10

■ As in logistic regression, gradient of the likelihood w.r.t. parameters is difference

between observed and expected feature counts:

■ Gradients can be computed by automatic differentiation! ■ In the Olden Days, would use the forward-backward algorithm to compute

expected counts.

count of feature j for token sequence w(i), tag sequence y(i)

` ✓j =

N

X

i=1

E[fj(w(i), y)] − fj(w(i), y(i))

Learning in CRFs

10

■ As in logistic regression, gradient of the likelihood w.r.t. parameters is difference

between observed and expected feature counts:

■ Gradients can be computed by automatic differentiation! ■ In the Olden Days, would use the forward-backward algorithm to compute

expected counts.

count of feature j for token sequence w(i), tag sequence y(i)

` ✓j =

N

X

i=1

E[fj(w(i), y)] − fj(w(i), y(i))

Ym1 = k0 Ym = k

αm1(k0) exp sm(k, k0) βm(k)

Learning in CRFs

10

■ As in logistic regression, gradient of the likelihood w.r.t. parameters is difference

between observed and expected feature counts:

■ Gradients can be computed by automatic differentiation! ■ In the Olden Days, would use the forward-backward algorithm to compute

expected counts.

count of feature j for token sequence w(i), tag sequence y(i)

` ✓j =

N

X

i=1

E[fj(w(i), y)] − fj(w(i), y(i))

Ym1 = k0 Ym = k

αm1(k0) exp sm(k, k0) βm(k)

forward score: sum over all prefixes

Learning in CRFs

10

■ As in logistic regression, gradient of the likelihood w.r.t. parameters is difference

between observed and expected feature counts:

■ Gradients can be computed by automatic differentiation! ■ In the Olden Days, would use the forward-backward algorithm to compute

expected counts.

count of feature j for token sequence w(i), tag sequence y(i)

` ✓j =

N

X

i=1

E[fj(w(i), y)] − fj(w(i), y(i))

Ym1 = k0 Ym = k

αm1(k0) exp sm(k, k0) βm(k)

forward score: sum over all prefixes backward score: sum over all suffixes

Learning in CRFs

10

■ As in logistic regression, gradient of the likelihood w.r.t. parameters is difference

between observed and expected feature counts:

■ Gradients can be computed by automatic differentiation! ■ In the Olden Days, would use the forward-backward algorithm to compute

expected counts.

count of feature j for token sequence w(i), tag sequence y(i)

` ✓j =

N

X

i=1

E[fj(w(i), y)] − fj(w(i), y(i))

Ym1 = k0 Ym = k

αm1(k0) exp sm(k, k0) βm(k)

forward score: sum over all prefixes backward score: sum over all suffixes transition score

Better features for sequence labeling?

11

■ Until now: hand-engineered features:

Better features for sequence labeling?

11

■ Until now: hand-engineered features:

wi contains a particular prefix (from all prefixes of length  4) wi contains a particular suffix (from all suffixes of length  4) wi contains a number wi contains an upper-case letter wi contains a hyphen wi is all upper case wi’s word shape wi’s short word shape wi is upper case and has a digit and a dash (like CFC-12) wi is upper case and followed within 3 words by Co., Inc., etc.

Lexical fi±{0,1,2,3}, (mi−2,i−1), (mi−1,i), (mi−1,i+1), (mi,i+1), (mi+1,i+2), (mi−2,i−1,i), (mi−1,i,i+1), (mi,i+1,i+2), (mi−2,i−1,i+1), (mi−1,i+1,i+2) POS pi−{3,2,1}, ai+{0,1,2,3}, (pi−2,i−1), (ai+1,i+2), (pi−1, ai+1), (pi−2, pi−1, ai), (pi−2, pi−1, ai+1), (pi−1, ai, ai+1), (pi−1, ai+1, ai+2) Affix c:1, c:2, c:3, cn:, cn−1:, cn−2:, cn−3: Binary initial uppercase, all uppercase/lowercase, contains 1/2+ capital(s) not at the beginning, contains a (period/number/hyphen)

Better features for sequence labeling?

11

■ Until now: hand-engineered features:

1 … 1 … … 1 …

wm-1 = will wm-1 = my wm = back wm+1 = ache ym-1 = MD

wi contains a particular prefix (from all prefixes of length  4) wi contains a particular suffix (from all suffixes of length  4) wi contains a number wi contains an upper-case letter wi contains a hyphen wi is all upper case wi’s word shape wi’s short word shape wi is upper case and has a digit and a dash (like CFC-12) wi is upper case and followed within 3 words by Co., Inc., etc.

Janet will back the bill .

Lexical fi±{0,1,2,3}, (mi−2,i−1), (mi−1,i), (mi−1,i+1), (mi,i+1), (mi+1,i+2), (mi−2,i−1,i), (mi−1,i,i+1), (mi,i+1,i+2), (mi−2,i−1,i+1), (mi−1,i+1,i+2) POS pi−{3,2,1}, ai+{0,1,2,3}, (pi−2,i−1), (ai+1,i+2), (pi−1, ai+1), (pi−2, pi−1, ai), (pi−2, pi−1, ai+1), (pi−1, ai, ai+1), (pi−1, ai+1, ai+2) Affix c:1, c:2, c:3, cn:, cn−1:, cn−2:, cn−3: Binary initial uppercase, all uppercase/lowercase, contains 1/2+ capital(s) not at the beginning, contains a (period/number/hyphen)

f(w, ym, ym−1, m) =

Better features for sequence labeling?

12

■ Until now: hand-engineered features:

1 … 1 … … 1 …

wm-1 = will wm-1 = my wm = back wm+1 = ache ym-1 = MD

Janet will back the bill .

f(w, ym, ym−1, m) =

pros: cons:

Better features for sequence labeling?

12

■ Until now: hand-engineered features:

1 … 1 … … 1 …

wm-1 = will wm-1 = my wm = back wm+1 = ache ym-1 = MD

Janet will back the bill .

f(w, ym, ym−1, m) =

pros: cons:

■ interpretable, explainable ■ can generalize well ■ fast training and inference ■ channel domain knowledge

Better features for sequence labeling?

12

■ Until now: hand-engineered features:

1 … 1 … … 1 …

wm-1 = will wm-1 = my wm = back wm+1 = ache ym-1 = MD

Janet will back the bill .

f(w, ym, ym−1, m) =

pros: cons:

■ can be sparse/high variance ■ lack of shared representations ■ task-specific ■ worse performance ■ interpretable, explainable ■ can generalize well ■ fast training and inference ■ channel domain knowledge

. . . . . .

Neural sequence labeling

13

■ Parameterize f with a (deep) neural network.

f(w, ym, ym−1, m) =

. . . . . .

Neural sequence labeling

13

■ Parameterize f with a (deep) neural network.

f(w, ym, ym−1, m) =

pros: cons:

. . . . . .

Neural sequence labeling

13

■ Parameterize f with a (deep) neural network.

f(w, ym, ym−1, m) =

pros: cons:

■ shared representations ■ channel external knowledge (e.g.

word embeddings)

■ high accuracy

. . . . . .

Neural sequence labeling

13

■ Parameterize f with a (deep) neural network.

f(w, ym, ym−1, m) =

pros: cons:

■ hard to interpret feature

meaning, explain predictions

■ optimization/hyperparameters ■ prone to overfitting ■ compute-heavy ■ shared representations ■ channel external knowledge (e.g.

word embeddings)

■ high accuracy

Neural sequence labeling

14

Neural sequence labeling

14

back the bill will Janet <s> </s>

word embeddings

Neural sequence labeling

14

back the bill will Janet <s> </s>

word embeddings

neural network

Neural sequence labeling

14

back the bill will Janet <s> </s>

word embeddings per-token features

neural network

Neural sequence labeling

14

back the bill will Janet <s> </s>

word embeddings per-token features

NNP MD VB? DT? NN?

neural network

Neural sequence labeling

15

Bidirectional RNNs

back the bill will Janet <s> </s>

word embeddings

Neural sequence labeling

15

Bidirectional RNNs

back the bill will Janet <s> </s>

word embeddings forward RNN

Neural sequence labeling

15

Bidirectional RNNs

back the bill will Janet <s> </s>

word embeddings forward RNN backward RNN

Neural sequence labeling

15

Bidirectional RNNs

concatenate

back the bill will Janet <s> </s>

word embeddings forward RNN backward RNN per-token features

Neural sequence labeling

15

Bidirectional RNNs

softmax(

)

concatenate

back the bill will Janet <s> </s>

word embeddings forward RNN backward RNN per-token features

V B N N A D J P R P W H M D D T

Neural sequence labeling

16

Bidirectional RNNs

back the bill will Janet <s> </s>

word embeddings forward RNN backward RNN per-token features

NNP MD VB?

Neural sequence labeling

16

Bidirectional RNNs

back the bill will Janet <s> </s>

word embeddings forward RNN backward RNN per-token features

NNP MD VB?

ti-1

Neural sequence labeling

17

Bidirectional RNNs

back the bill will Janet <s> </s>

word embeddings forward RNN backward RNN per-token features

NNP MD VB?

ti-1

DT? NN?

Neural sequence labeling

17

Bidirectional RNNs

back the bill will Janet <s> </s>

word embeddings forward RNN backward RNN per-token features

NNP MD VB? DT? NN?

Neural sequence labeling

18

Bidirectional RNNs

back the bill will Janet <s> </s>

word embeddings forward RNN backward RNN per-token features

NNP MD VB? DT? NN?

Neural sequence labeling

18

Bidirectional RNNs

back the bill will Janet <s> </s>

word embeddings forward RNN backward RNN per-token features

NNP MD VB? DT? NN?

Neural sequence labeling

19

Bidirectional RNN-CRFs

back the bill will Janet <s> </s>

word embeddings forward RNN backward RNN per-token features

NNP MD VB? DT? NN?

Neural sequence labeling

20

back the bill will Janet <s> </s>

word embeddings per-token features

NNP MD VB? DT? NN?

neural network

Convolutional neural networks

21

Convolutional neural networks

21

■ In NLP

, CNNs merge information across contiguous, fixed-width spans of tokens.

Convolutional neural networks

21

■ In NLP

, CNNs merge information across contiguous, fixed-width spans of tokens.

■ Unlike computer vision, in NLP we use 1D CNNs.

Convolutional neural networks

21

■ In NLP

, CNNs merge information across contiguous, fixed-width spans of tokens.

■ Unlike computer vision, in NLP we use 1D CNNs. ■ For sentence/document classification: pooling function over representations.

wait for the video and do n't rent it

n x k representation of sentence with static and non-static channels Convolutional layer with multiple filter widths and feature maps Max-over-time pooling Fully connected layer with dropout and softmax output

Figure from: Yoon Kim. Convolutional Neural Networks for Sentence Classification. EMNLP 2014.

Convolutional neural networks

21

■ In NLP

, CNNs merge information across contiguous, fixed-width spans of tokens.

■ Unlike computer vision, in NLP we use 1D CNNs. ■ For sentence/document classification: pooling function over representations. ■ For example: sum, average. Most common: max pooling (over time).

wait for the video and do n't rent it

n x k representation of sentence with static and non-static channels Convolutional layer with multiple filter widths and feature maps Max-over-time pooling Fully connected layer with dropout and softmax output

Figure from: Yoon Kim. Convolutional Neural Networks for Sentence Classification. EMNLP 2014.

Convolutional neural networks

22

■ In NLP

, CNNs merge information across contiguous, fixed-width spans of tokens. back the bill will Janet <s>

dims: [dword]

Convolutional neural networks

22

■ In NLP

, CNNs merge information across contiguous, fixed-width spans of tokens. back the bill will Janet <s>

dims: [dword] kernel size = 3

Convolutional neural networks

22

■ In NLP

, CNNs merge information across contiguous, fixed-width spans of tokens. back the bill will Janet <s>

dims: [dword] [kdword] kernel size = 3

Convolutional neural networks

22

■ In NLP

, CNNs merge information across contiguous, fixed-width spans of tokens. back the bill will Janet <s>

Θ(x→z)

dims: [kdword x dz] [dz] [dword] [kdword] kernel size = 3 # filters

Convolutional neural networks

23

back the bill will Janet <s>

dims: [kdword x dz] [dz] [dword] [kdword]

Θ(x→z)

■ In NLP

, CNNs merge information across contiguous, fixed-width spans of tokens.

Convolutional neural networks

23

back the bill will Janet <s>

dims: [kdword x dz] [dz] [dword] [kdword]

Θ(x→z)

stride = 1

■ In NLP

, CNNs merge information across contiguous, fixed-width spans of tokens.

Convolutional neural networks

23

back the bill will Janet <s>

dims: [kdword x dz] [dz] [dword] [kdword]

Θ(x→z)

stride = 1

■ In NLP

, CNNs merge information across contiguous, fixed-width spans of tokens.

Convolutional neural networks

23

back the bill will Janet <s>

dims: [kdword x dz] [dz] [dword] [kdword]

Θ(x→z)

stride = 1

■ In NLP

, CNNs merge information across contiguous, fixed-width spans of tokens.

Convolutional neural networks

23

back the bill will Janet <s>

dims: [kdword x dz] [dz] [dword] [kdword]

Θ(x→z)

stride = 1

■ In NLP

, CNNs merge information across contiguous, fixed-width spans of tokens.

# filters

Sequence labeling w/ CNNs

24

committee awards Strickland advanced optics who Nobel

I-ORG B-PER B-ORG O O O O

Sequence labeling w/ CNNs

24

committee awards Strickland advanced optics who Nobel

I-ORG B-PER B-ORG O O O O

Sequence labeling w/ CNNs

24

committee awards Strickland advanced optics who Nobel

I-ORG B-PER B-ORG O O O O

Sequence labeling w/ CNNs

24

committee awards Strickland advanced optics who Nobel

I-ORG B-PER B-ORG O O O O

Sequence labeling w/ CNNs

24

committee awards Strickland advanced optics who Nobel

I-ORG B-PER B-ORG O O O O

Sequence labeling w/ CNNs

24

committee awards Strickland advanced optics who Nobel

I-ORG B-PER B-ORG O O O O

Sequence labeling w/ CNNs

24

committee awards Strickland advanced optics who Nobel

I-ORG B-PER B-ORG O O O O

Sequence labeling w/ CNNs

24

committee awards Strickland advanced optics who Nobel

I-ORG B-PER B-ORG O O O O

encode in parallel

Sequence labeling w/ CNNs

24

committee awards Strickland advanced optics who Nobel

I-ORG B-PER B-ORG O O O O

encode in parallel

Sequence labeling w/ CNNs

24

committee awards Strickland advanced optics who Nobel

I-ORG B-PER B-ORG O O O O

encode in parallel

■ Used for semantic role labeling, with poor results [Collobert et al. 2011].

Sequence labeling w/ CNNs

24

committee awards Strickland advanced optics who Nobel

I-ORG B-PER B-ORG O O O O

encode in parallel

■ Used for semantic role labeling, with poor results [Collobert et al. 2011].

Sequence labeling w/ CNNs

24

committee awards Strickland advanced optics who Nobel

I-ORG B-PER B-ORG O O O O

encode in parallel

■ Used for semantic role labeling, with poor results [Collobert et al. 2011]. ■ Not enough context: amount of context grows linearly w/ number of layers.

Sequence labeling w/ dilated CNNs

25

committee awards Strickland advanced optics who Nobel

Sequence labeling w/ dilated CNNs

25

■ Additional parameter: dilation width δ

committee awards Strickland advanced optics who Nobel

Sequence labeling w/ dilated CNNs

25

■ Additional parameter: dilation width δ

committee awards Strickland advanced optics who Nobel

δ=1

Sequence labeling w/ dilated CNNs

25

■ Additional parameter: dilation width δ

committee awards Strickland advanced optics who Nobel

δ=1

Sequence labeling w/ dilated CNNs

25

■ Additional parameter: dilation width δ

committee awards Strickland advanced optics who Nobel

δ=1 δ=2

Sequence labeling w/ dilated CNNs

25

■ Additional parameter: dilation width δ

committee awards Strickland advanced optics who Nobel

δ=1 δ=2

Sequence labeling w/ dilated CNNs

25

■ Additional parameter: dilation width δ

committee awards Strickland advanced optics who Nobel

δ=1 δ=2 δ=4

Sequence labeling w/ dilated CNNs

25

■ Additional parameter: dilation width δ

committee awards Strickland advanced optics who Nobel

δ=1 δ=2 δ=4

Sequence labeling w/ dilated CNNs

25

■ Additional parameter: dilation width δ

committee awards Strickland advanced optics who Nobel

δ=1 δ=2 δ=4

Sequence labeling w/ dilated CNNs

25

■ Additional parameter: dilation width δ

committee awards Strickland advanced optics who Nobel

δ=1 δ=2 δ=4 δ=1

Sequence labeling w/ dilated CNNs

25

■ Additional parameter: dilation width δ

committee awards Strickland advanced optics who Nobel

B-ORG B-PER I-ORG

O O O O δ=1 δ=2 δ=4 δ=1

Sequence labeling w/ dilated CNNs

25

■ Additional parameter: dilation width δ ■ Context window grows exponentially w/ number of layers.

committee awards Strickland advanced optics who Nobel

B-ORG B-PER I-ORG

O O O O δ=1 δ=2 δ=4 δ=1

Sequence labeling w/ dilated CNNs

26

Sequence labeling w/ dilated CNNs

26

■ Why use a (dilated) CNN over a (bidirectional) LSTM?

Sequence labeling w/ dilated CNNs

26

■ Why use a (dilated) CNN over a (bidirectional) LSTM? ■ Efficiency (on GPUs). Representations for every token in the sequence can be

computed in parallel for CNN; linear dependence on sequence length for LSTM.

Sequence labeling w/ dilated CNNs

26

■ Why use a (dilated) CNN over a (bidirectional) LSTM? ■ Efficiency (on GPUs). Representations for every token in the sequence can be

computed in parallel for CNN; linear dependence on sequence length for LSTM.

NER F1 sentence document Bi-LSTM-CRF 90.4 ± 0.1 90.6 ± 0.2 ID-CNN (ours) 90.3 ± 0.3 90.7 ± 0.2

Sequence labeling w/ dilated CNNs

26

■ Why use a (dilated) CNN over a (bidirectional) LSTM? ■ Efficiency (on GPUs). Representations for every token in the sequence can be

computed in parallel for CNN; linear dependence on sequence length for LSTM.

NER F1 sentence document Bi-LSTM-CRF 90.4 ± 0.1 90.6 ± 0.2 ID-CNN (ours) 90.3 ± 0.3 90.7 ± 0.2

Sequence labeling w/ dilated CNNs

26

■ Why use a (dilated) CNN over a (bidirectional) LSTM? ■ Efficiency (on GPUs). Representations for every token in the sequence can be

computed in parallel for CNN; linear dependence on sequence length for LSTM.

NER F1 sentence document Bi-LSTM-CRF 90.4 ± 0.1 90.6 ± 0.2 ID-CNN (ours) 90.3 ± 0.3 90.7 ± 0.2

1 4 x s p e e d

• u

p

Sequence labeling w/ dilated CNNs

26

■ Why use a (dilated) CNN over a (bidirectional) LSTM? ■ Efficiency (on GPUs). Representations for every token in the sequence can be

computed in parallel for CNN; linear dependence on sequence length for LSTM.

NER F1 sentence document Bi-LSTM-CRF 90.4 ± 0.1 90.6 ± 0.2 ID-CNN (ours) 90.3 ± 0.3 90.7 ± 0.2

1 4 x s p e e d

• u

p

Sequence labeling w/ dilated CNNs

26

■ Why use a (dilated) CNN over a (bidirectional) LSTM? ■ Efficiency (on GPUs). Representations for every token in the sequence can be

computed in parallel for CNN; linear dependence on sequence length for LSTM.

NER F1 sentence document Bi-LSTM-CRF 90.4 ± 0.1 90.6 ± 0.2 ID-CNN (ours) 90.3 ± 0.3 90.7 ± 0.2

1 4 x s p e e d

• u

p 8 x s p e e d

• u

p

Character embeddings

27

Character embeddings

27

■ Character-level representations of words help to deal with UNKs.

Character embeddings

27

■ Character-level representations of words help to deal with UNKs. ■ Usually, CNNs + pooling are used to compose characters into word embeddings.

Character embeddings

27

■ Character-level representations of words help to deal with UNKs. ■ Usually, CNNs + pooling are used to compose characters into word embeddings.

Images from: Ma and Hovy. End-to-end Sequence Labeling via Bi-directional LSTM-CNNs-CRF . ACL 2016.
 Lample et al. Neural Architectures for Named Entity Recognition. NAACL 2016.

P l a y i n g Padding Padding Char Embedding Convolution Max Pooling Char Representation

CNN + max pooling

Character embeddings

27

■ Character-level representations of words help to deal with UNKs. ■ Usually, CNNs + pooling are used to compose characters into word embeddings.

Images from: Ma and Hovy. End-to-end Sequence Labeling via Bi-directional LSTM-CNNs-CRF . ACL 2016.
 Lample et al. Neural Architectures for Named Entity Recognition. NAACL 2016.

P l a y i n g Padding Padding Char Embedding Convolution Max Pooling Char Representation

bidirectional LSTM CNN + max pooling

Character embeddings

27

■ Character-level representations of words help to deal with UNKs. ■ Usually, CNNs + pooling are used to compose characters into word embeddings.

Images from: Ma and Hovy. End-to-end Sequence Labeling via Bi-directional LSTM-CNNs-CRF . ACL 2016.
 Lample et al. Neural Architectures for Named Entity Recognition. NAACL 2016.

P l a y i n g Padding Padding Char Embedding Convolution Max Pooling Char Representation

bidirectional LSTM CNN + max pooling

Multilingual part-of-speech tagging

28

Multilingual part-of-speech tagging

28

■ Many UNKs in morphologically-rich languages like Czech, Hungarian, Turkish

Multilingual part-of-speech tagging

28

■ Many UNKs in morphologically-rich languages like Czech, Hungarian, Turkish ■ 250,000 word corpus of Hungarian has > 2x as many types as a similarly

sized corpus of English

Multilingual part-of-speech tagging

28

■ Many UNKs in morphologically-rich languages like Czech, Hungarian, Turkish ■ 250,000 word corpus of Hungarian has > 2x as many types as a similarly

sized corpus of English

■ 10 million word corpus of Turkish contains 4x as many types

Multilingual part-of-speech tagging

28

■ Many UNKs in morphologically-rich languages like Czech, Hungarian, Turkish ■ 250,000 word corpus of Hungarian has > 2x as many types as a similarly

sized corpus of English

■ 10 million word corpus of Turkish contains 4x as many types ■ Information coded in morphology

Multilingual part-of-speech tagging

28

■ Many UNKs in morphologically-rich languages like Czech, Hungarian, Turkish ■ 250,000 word corpus of Hungarian has > 2x as many types as a similarly

sized corpus of English

■ 10 million word corpus of Turkish contains 4x as many types ■ Information coded in morphology

partilerindeydi partisindeydiler

Multilingual part-of-speech tagging

28

■ Many UNKs in morphologically-rich languages like Czech, Hungarian, Turkish ■ 250,000 word corpus of Hungarian has > 2x as many types as a similarly

sized corpus of English

■ 10 million word corpus of Turkish contains 4x as many types ■ Information coded in morphology

partilerindeydi partisindeydiler

he/she/they(sing) were/was at their(plur) party

Multilingual part-of-speech tagging

28

■ Many UNKs in morphologically-rich languages like Czech, Hungarian, Turkish ■ 250,000 word corpus of Hungarian has > 2x as many types as a similarly

sized corpus of English

■ 10 million word corpus of Turkish contains 4x as many types ■ Information coded in morphology

partilerindeydi partisindeydiler

he/she/they(sing) were/was at their(plur) party they(plur) were at his/her/their(sing) party

Multilingual part-of-speech tagging

28

■ Many UNKs in morphologically-rich languages like Czech, Hungarian, Turkish ■ 250,000 word corpus of Hungarian has > 2x as many types as a similarly

sized corpus of English

■ 10 million word corpus of Turkish contains 4x as many types ■ Information coded in morphology

partilerindeydi partisindeydiler

he/she/they(sing) were/was at their(plur) party party party they(plur) were at his/her/their(sing) party

Multilingual part-of-speech tagging

28

■ Many UNKs in morphologically-rich languages like Czech, Hungarian, Turkish ■ 250,000 word corpus of Hungarian has > 2x as many types as a similarly

sized corpus of English

■ 10 million word corpus of Turkish contains 4x as many types ■ Information coded in morphology

partilerindeydi partisindeydiler

he/she/they(sing) were/was at their(plur) party party in party in they(plur) were at his/her/their(sing) party

Multilingual part-of-speech tagging

28

■ Many UNKs in morphologically-rich languages like Czech, Hungarian, Turkish ■ 250,000 word corpus of Hungarian has > 2x as many types as a similarly

sized corpus of English

■ 10 million word corpus of Turkish contains 4x as many types ■ Information coded in morphology

partilerindeydi partisindeydiler

he/she/they(sing) were/was at their(plur) party party their in she/he/they(sing) was party his/her/their(sing) in they(plur) were they(plur) were at his/her/their(sing) party

Multilingual part-of-speech tagging

28

■ Many UNKs in morphologically-rich languages like Czech, Hungarian, Turkish ■ 250,000 word corpus of Hungarian has > 2x as many types as a similarly

sized corpus of English

■ 10 million word corpus of Turkish contains 4x as many types ■ Information coded in morphology

partilerindeydi partisindeydiler

he/she/they(sing) were/was at their(plur) party party their in she/he/they(sing) was party his/her/their(sing) in they(plur) were they(plur) were at his/her/their(sing) party

• 1. Yerdeki izin temizlenmesi gerek.

iz + Noun+A3sg+Pnon+Gen The trace on the floor should be cleaned. ¨

Multilingual part-of-speech tagging

29

Multilingual part-of-speech tagging

29

■ In non-word-space languages like Chinese, word segmentation is either applied

before tagging or performed jointly.

Multilingual part-of-speech tagging

29

■ In non-word-space languages like Chinese, word segmentation is either applied

before tagging or performed jointly.

⁄€e;≥[ “Yao Ming reaches

Yao Ming reaches the finals

Multilingual part-of-speech tagging

29

■ In non-word-space languages like Chinese, word segmentation is either applied

before tagging or performed jointly.

⁄€e;≥[ “Yao Ming reaches

Yao Ming reaches the finals

⁄ “Yao ;≥[ Ming reaches €e; ao Ming

YaoMing reaches finals

Multilingual part-of-speech tagging

29

■ In non-word-space languages like Chinese, word segmentation is either applied

before tagging or performed jointly.

⁄€e;≥[ “Yao Ming reaches

Yao Ming reaches the finals

⁄ “Yao ;≥[ Ming reaches €e; ao Ming

YaoMing reaches finals CTB

Multilingual part-of-speech tagging

29

■ In non-word-space languages like Chinese, word segmentation is either applied

before tagging or performed jointly.

⁄€e;≥[ “Yao Ming reaches

Yao Ming reaches the finals

⁄ “Yao ;≥[ Ming reaches €e; ao Ming

YaoMing reaches finals CTB

⁄ “Yao ;≥[ reaches €e; ao Ming

Yao Ming reaches overall finals

;≥ Ming reaches

• “Yao

Multilingual part-of-speech tagging

29

■ In non-word-space languages like Chinese, word segmentation is either applied

before tagging or performed jointly.

⁄€e;≥[ “Yao Ming reaches

Yao Ming reaches the finals

⁄ “Yao ;≥[ Ming reaches €e; ao Ming

YaoMing reaches finals CTB

⁄ “Yao ;≥[ reaches €e; ao Ming

Yao Ming reaches overall finals

;≥ Ming reaches

Peking U.

• “Yao

Multilingual part-of-speech tagging

29

■ In non-word-space languages like Chinese, word segmentation is either applied

before tagging or performed jointly.

■ UNKs are difficult: majority of unknown words are

common nouns and verbs due to compounding

⁄€e;≥[ “Yao Ming reaches

Yao Ming reaches the finals

⁄ “Yao ;≥[ Ming reaches €e; ao Ming

YaoMing reaches finals CTB

⁄ “Yao ;≥[ reaches €e; ao Ming

Yao Ming reaches overall finals

;≥ Ming reaches

Peking U.

• “Yao

Multilingual part-of-speech tagging

29

■ In non-word-space languages like Chinese, word segmentation is either applied

before tagging or performed jointly.

■ UNKs are difficult: majority of unknown words are

common nouns and verbs due to compounding

夏 天 太 热 (too) (hot) (summer) character representations GRU GRU GRU GRU forward RNN GRU GRU GRU GRU backward RNN B-NT E-NT S-AD S-VA CRF Layer 太 AD 热 VA 夏天 NT Output

Figure from: Shao et al. Character-based Joint Segmentation and POS Tagging for Chinese using Bidirectional RNN-CRF . IJCNLP 2017.

⁄€e;≥[ “Yao Ming reaches

Yao Ming reaches the finals I O B E S t a g g i n g

⁄ “Yao ;≥[ Ming reaches €e; ao Ming

YaoMing reaches finals CTB

⁄ “Yao ;≥[ reaches €e; ao Ming

Yao Ming reaches overall finals

;≥ Ming reaches

Peking U.

• “Yao

Multilingual part-of-speech tagging

30

Multilingual part-of-speech tagging

30

■ Universal POS tags [Petrov et al. 2012] provide a cross-lingual tag set.

Language Source # Tags Arabic PADT/CoNLL07 (Hajiˇ c et al., 2004) 21 Basque Basque3LB/CoNLL07 (Aduriz et al., 2003) 64 Bulgarian BTB/CoNLL06 (Simov et al., 2002) 54 Catalan CESS-ECE/CoNLL07 (Mart´ ı et al., 2007) 54 Chinese Penn ChineseTreebank 6.0 (Palmer et al., 2007) 34 Chinese Sinica/CoNLL07 (Chen et al., 2003) 294 Czech PDT/CoNLL07 (B¨

• hmov´

a et al., 2003) 63 Danish DDT/CoNLL06 (Kromann et al., 2003) 25 Dutch Alpino/CoNLL06 (Van der Beek et al., 2002) 12 English PennTreebank (Marcus et al., 1993) 45 French FrenchTreebank (Abeill´ e et al., 2003) 30 German Tiger/CoNLL06 (Brants et al., 2002) 54 German Negra (Skut et al., 1997) 54 Greek GDT/CoNLL07 (Prokopidis et al., 2005) 38 Hungarian Szeged/CoNLL07 (Csendes et al., 2005) 43 Italian ISST/CoNLL07 (Montemagni et al., 2003) 28 Japanese Verbmobil/CoNLL06 (Kawata and Bartels, 2000) 80 Japanese Kyoto4.0 (Kurohashi and Nagao, 1997) 42 Korean Sejong (http://www.sejong.or.kr) 187 Portuguese Floresta Sint´ a(c)tica/CoNLL06 (Afonso et al., 2002) 22 Russian SynTagRus-RNC (Boguslavsky et al., 2002) 11 Slovene SDT/CoNLL06 (Dˇ zeroski et al., 2006) 29 Spanish Ancora-Cast3LB/CoNLL06 (Civit and Mart´ ı, 2004) 47 Swedish Talbanken05/CoNLL06 (Nivre et al., 2006) 41 Turkish METU-Sabanci/CoNLL07 (Oflazer et al., 2003) 31

Table from: Petrov, Das and McDonald. A Universal Part-of-Speech Tagset. LREC 2012.

Multilingual part-of-speech tagging

30

■ Universal POS tags [Petrov et al. 2012] provide a cross-lingual tag set. ■ Coarse grained: 16 tags

• pen class

closed class

• ther

ADJ ADP PUNCT ADV AUX SYM INTJ CCONJ X NOUN DET PROPN NUM VERB PART PRON SCONJ

Multilingual part-of-speech tagging

30

■ Universal POS tags [Petrov et al. 2012] provide a cross-lingual tag set. ■ Coarse grained: 16 tags ■ Finer-grained analysis split off into morphological tags (case, gender, …)

• pen class

closed class

• ther

ADJ ADP PUNCT ADV AUX SYM INTJ CCONJ X NOUN DET PROPN NUM VERB PART PRON SCONJ

Multilingual part-of-speech tagging

30

■ Universal POS tags [Petrov et al. 2012] provide a cross-lingual tag set. ■ Coarse grained: 16 tags ■ Finer-grained analysis split off into morphological tags (case, gender, …)

Example from: https://universaldependencies.org/

English: Bulgarian: Czech: Swedish:

• pen class

closed class

• ther

ADJ ADP PUNCT ADV AUX SYM INTJ CCONJ X NOUN DET PROPN NUM VERB PART PRON SCONJ

Announcements

31

■ Project 2 released today after class: sequence labeling. ■ Due: October 16. ■ You will implement part-of-speech taggers for English and Norwegian: ■ HMM, BiLSTM, and BiLSTM-CRF. ■ Friday’s recitation will be an overview of P2.