Algorithms for NLP CS 11-711 Fall 2020 Lecture 3: Nonlinear text - - PowerPoint PPT Presentation

Emma Strubell

Algorithms for NLP

CS 11-711 · Fall 2020

Lecture 3: Nonlinear text classification

Announcements

2

■ Project 1: Text classification will be available after class today,

due Friday September 25

■ Han will lead recitation this Friday to introduce Project 1, and cover environment

setup (Python, Jupyter, NumPy, PyTorch).

Recap

Representing text as a bag of words

3

■ Given a text ■ Choose a label ■ The bag-of-words is a fixed-length vector of word counts: ■ Length of x is equal to the size of the vocabulary, V

xt w = (w1, w2, . . . , wT) ∈ V∗

el y ∈ Y.

The drinks were strong but the fish tacos were bland

w = x =

a a r d v a r k

… 1 1 … 1 … 1 2 1 … 2 …

b u t t h e w e r e b l a n d z y t h e r … t a c

• s

s t r

• n

g t a c

• fi

s h … … … …

Recap

4

■ Let score the compatibility of bag-of-words x and label y.

Then:

■ In a linear classifier this scoring function has the simple form:

where θ is a vector of weights, and f is a feature function

ˆ y = argmax

y

ψ(x, y).

x ψ(x, y).

ψ(x, y) = θ · f (x, y) = X

j=1

θj × fj(x, y),

Linear classification on bag-of-words

Recap

5

■ In classification, the feature function is usually a simple combination of x and y,

such as:

■ If we have K labels, this corresponds to column vectors that look like:

fj(x, y) = ( x

xbland if y = negative

• therwise

f (x, y = 1) = f (x, y = 2) = [

x0 x1 … x|V| … … x0 x1 … x|V| …

f (x, y = K) =

… x0 x1 … x|V|

| {z }

(K−1)×V

T T T

Feature functions

| {z }

(K−2)×V

{z

V

| {z }

(K−1)×V

How to obtain θ ?

6

How to obtain θ ?

6

■ The learning problem is to find the right weights θ.

How to obtain θ ?

6

■ The learning problem is to find the right weights θ. ■ Naïve Bayes: set θ equal to the empirical frequencies:

P = count(y) P

y0 count(y0).

= count(y, j) PV

j0=1 count(y, j0)

ˆ µy = p(y) =

ˆ φy,j = p(xj | y) =

How to obtain θ ?

6

■ The learning problem is to find the right weights θ. ■ Naïve Bayes: set θ equal to the empirical frequencies: ■ Perceptron update:

θ + f (x(i), y(i)) − f (x(i), ˆ y).

θ(t+1) ←θ(t) −

P = count(y) P

y0 count(y0).

= count(y, j) PV

j0=1 count(y, j0)

ˆ µy = p(y) =

ˆ φy,j = p(xj | y) =

How to obtain θ ?

6

■ The learning problem is to find the right weights θ. ■ Naïve Bayes: set θ equal to the empirical frequencies: ■ Perceptron update: ■ Large-margin update:

θ + f (x(i), y(i)) − f (x(i), ˆ y).

θ(t+1) ←θ(t) −

P = count(y) P

y0 count(y0).

= count(y, j) PV

j0=1 count(y, j0)

with

θ(t+1) ← θ(t) + f(x(i), y (i)) − f(x(i), ˆ y)

ˆ y = argmax

y∈Y

θ · f(x(i), y) + c(y (i), y)

ˆ µy = p(y) =

ˆ φy,j = p(xj | y) =

How to obtain θ ?

6

■ The learning problem is to find the right weights θ. ■ Naïve Bayes: set θ equal to the empirical frequencies: ■ Perceptron update: ■ Large-margin update: ■ Logistic regression update:

θ + f (x(i), y(i)) − f (x(i), ˆ y).

θ(t+1) ←θ(t) −

P = count(y) P

y0 count(y0).

= count(y, j) PV

j0=1 count(y, j0)

with

θ(t+1) ← θ(t) + f(x(i), y (i)) − f(x(i), ˆ y)

ˆ y = argmax

y∈Y

θ · f(x(i), y) + c(y (i), y)

ˆ µy = p(y) =

ˆ φy,j = p(xj | y) =

θ(t+1) ← θ(t) + f(x(i), y (i)) − Ey|x h f(x(i), y) i

How to obtain θ ?

6

■ The learning problem is to find the right weights θ. ■ Naïve Bayes: set θ equal to the empirical frequencies: ■ Perceptron update: ■ Large-margin update: ■ Logistic regression update: ■ All these methods for supervised learning assume a labeled dataset of N examples:

aset {(x(i), y(i))}N

i=1.

θ + f (x(i), y(i)) − f (x(i), ˆ y).

θ(t+1) ←θ(t) −

P = count(y) P

y0 count(y0).

= count(y, j) PV

j0=1 count(y, j0)

with

θ(t+1) ← θ(t) + f(x(i), y (i)) − f(x(i), ˆ y)

ˆ y = argmax

y∈Y

θ · f(x(i), y) + c(y (i), y)

ˆ µy = p(y) =

ˆ φy,j = p(xj | y) =

θ(t+1) ← θ(t) + f(x(i), y (i)) − Ey|x h f(x(i), y) i

Today

7

Engineered features

Nonlinear classification & evaluating classifiers

Today

7

Engineered features linear classification

Nonlinear classification & evaluating classifiers

Today

7

~mid 2010s Engineered features Learned features linear classification

Nonlinear classification & evaluating classifiers

Today

7

~mid 2010s Engineered features Learned features linear classification nonlinear classification

Nonlinear classification & evaluating classifiers

A simple feed-forward architecture

8

A simple feed-forward architecture

■ Suppose we want to label stories as

8

s Y = {Good, Bad, Okay}.

A simple feed-forward architecture

■ Suppose we want to label stories as ■ What makes a good story?

8

s Y = {Good, Bad, Okay}.

A simple feed-forward architecture

■ Suppose we want to label stories as ■ What makes a good story? ■ Exciting plot, compelling characters, interesting setting…

8

s Y = {Good, Bad, Okay}.

A simple feed-forward architecture

■ Suppose we want to label stories as ■ What makes a good story? ■ Exciting plot, compelling characters, interesting setting… ■ Let’s call this vector of features z.

8

s Y = {Good, Bad, Okay}.

A simple feed-forward architecture

■ Suppose we want to label stories as ■ What makes a good story? ■ Exciting plot, compelling characters, interesting setting… ■ Let’s call this vector of features z. ■ If z is well-chosen, it will be easy to predict from x (the words), and it will make it

easy to predict the label, y.

8

s Y = {Good, Bad, Okay}.

A simple feed-forward architecture

9

. . . . . . x z y

A simple feed-forward architecture

■ Let’s predict each zk from x by binary logistic

regression:

9

. . . . . . x z y

Pr(zk = 1 | x) =σ(θ(x→z)

k

· x)

A simple feed-forward architecture

■ Let’s predict each zk from x by binary logistic

regression:

9

. . . . . . x z y

Pr(zk = 1 | x) =σ(θ(x→z)

k

· x)

logistic fn aka sigmoid σ

= 1 1 + e−θ(x→z)

k

x

A simple feed-forward architecture

■ Let’s predict each zk from x by binary logistic

regression:

9

. . . . . . x z y

Pr(zk = 1 | x) =σ(θ(x→z)

k

· x)

logistic fn aka sigmoid σ

= 1 1 + e−θ(x→z)

k

x

A simple feed-forward architecture

■ Let’s predict each zk from x by binary logistic

regression:

■ The weights can be collected into a matrix,

9

. . . . . . x z y

Pr(zk = 1 | x) =σ(θ(x→z)

k

· x) Θ(x!z) = [θ(x!z)

1

, θ(x!z)

2

, . . . , θ(x!z)

Kz

]>,

logistic fn aka sigmoid σ

= 1 1 + e−θ(x→z)

k

x

A simple feed-forward architecture

■ Let’s predict each zk from x by binary logistic

regression:

■ The weights can be collected into a matrix, ■ so that E[z] = σ(Θ(x→z)x), where σ is applied

element-wise.

9

. . . . . . x z y

Pr(zk = 1 | x) =σ(θ(x→z)

k

· x) Θ(x!z) = [θ(x!z)

1

, θ(x!z)

2

, . . . , θ(x!z)

Kz

]>,

logistic fn aka sigmoid σ

= 1 1 + e−θ(x→z)

k

x

A simple feed-forward architecture

■ Let’s predict each zk from x by binary logistic

regression:

■ The weights can be collected into a matrix, ■ so that E[z] = σ(Θ(x→z)x), where σ is applied

element-wise.

9

. . . . . . x z y

Pr(zk = 1 | x) =σ(θ(x→z)

k

· x) Θ(x!z) = [θ(x!z)

1

, θ(x!z)

2

, . . . , θ(x!z)

Kz

]>,

logistic fn aka sigmoid σ

= 1 1 + e−θ(x→z)

k

x

matrix-vector product. dims: [k, V] * [V, 1] = [k, 1]

A simple feed-forward architecture

10

. . . . . . x z y

A simple feed-forward architecture

■ Next we predict y from z, again using logistic

regression (multiclass): Vector of probabilities over each possible y is denoted:

10

. . . . . . x z y | = exp(θ(z→y)

j

· z + bj) P

j0∈Y exp(θ(z→y) j0

· z + bj0) ,

Pr(y = j | z)

A simple feed-forward architecture

■ Next we predict y from z, again using logistic

regression (multiclass): Vector of probabilities over each possible y is denoted:

10

. . . . . . x z y | = exp(θ(z→y)

j

· z + bj) P

j0∈Y exp(θ(z→y) j0

· z + bj0) ,

Pr(y = j | z)

additive bias/offset vector

A simple feed-forward architecture

■ Next we predict y from z, again using logistic

regression (multiclass): Vector of probabilities over each possible y is denoted:

10

. . . . . . x z y | = exp(θ(z→y)

j

· z + bj) P

j0∈Y exp(θ(z→y) j0

· z + bj0) ,

Pr(y = j | z)

additive bias/offset vector

p(y | z) = SoftMax(Θ(z→y)z + b).

A simple feed-forward architecture

11

. . . . . . x z y

A simple feed-forward architecture

11

■ In reality, we never observe z, it is a hidden layer. We don’t bother predicting 0/1

values for z, we compute it directly from x.

. . . . . . x z y

A simple feed-forward architecture

11

■ In reality, we never observe z, it is a hidden layer. We don’t bother predicting 0/1

values for z, we compute it directly from x.

■ This makes p(y | x) a complex, nonlinear function of x.

. . . . . . x z y

A simple feed-forward architecture

11

■ In reality, we never observe z, it is a hidden layer. We don’t bother predicting 0/1

values for z, we compute it directly from x.

■ This makes p(y | x) a complex, nonlinear function of x. ■ We can have multiple hidden layers z(1), z(2), … adding even more expressiveness.

. . . . . . x z y

A simple feed-forward architecture

■ To summarize:

where

11

p(y | z) = SoftMax(Θ(z→y)z + b).

z =σ(Θ(x→z)x)

σ(Θ(x→z)x) = h σ(θ(x→z)

1

· x), σ(θ(x→z)

2

· x), ..., σ(θ(x→z)

Kz

· x) iT

■ In reality, we never observe z, it is a hidden layer. We don’t bother predicting 0/1

values for z, we compute it directly from x.

■ This makes p(y | x) a complex, nonlinear function of x. ■ We can have multiple hidden layers z(1), z(2), … adding even more expressiveness.

. . . . . . x z y

Activation functions

12

Activation functions

12

■ The sigmoid in is called an activation function.

in z = σ(Θ(x→z)x) i

Activation functions

12

■ The sigmoid in is called an activation function. ■ In general, we can write to indicate an arbitrary activation function.

in z = σ(Θ(x→z)x) i

e z = f (Θ(x→z)x) t

Activation functions

12

■ The sigmoid in is called an activation function. ■ In general, we can write to indicate an arbitrary activation function. ■ Other choices include:

in z = σ(Θ(x→z)x) i

e z = f (Θ(x→z)x) t

Activation functions

12

■ The sigmoid in is called an activation function. ■ In general, we can write to indicate an arbitrary activation function. ■ Other choices include: ■ Hyperbolic tangent: tanh, centered at 0, helps avoid saturation

in z = σ(Θ(x→z)x) i

e z = f (Θ(x→z)x) t

Activation functions

12

■ The sigmoid in is called an activation function. ■ In general, we can write to indicate an arbitrary activation function. ■ Other choices include: ■ Hyperbolic tangent: tanh, centered at 0, helps avoid saturation ■ Rectified linear unit: ReLU(a) = max(0, a), which is fast to evaluate, easy to

analyze, even further avoids saturation.

in z = σ(Θ(x→z)x) i

e z = f (Θ(x→z)x) t

Activation functions

12

■ The sigmoid in is called an activation function. ■ In general, we can write to indicate an arbitrary activation function. ■ Other choices include: ■ Hyperbolic tangent: tanh, centered at 0, helps avoid saturation ■ Rectified linear unit: ReLU(a) = max(0, a), which is fast to evaluate, easy to

analyze, even further avoids saturation.

in z = σ(Θ(x→z)x) i

e z = f (Θ(x→z)x) t

■ Leaky ReLU:

) = ( a, a ≥ 0 .0001a,

• therwise.

Training neural networks

13

Gradient descent

Training neural networks

13

■ In general, neural networks are learned by gradient descent, using minibatches:

where

θ(z!y)

k

θ(z!y)

k

⌘(t)rθ(z→y)

k

`(i),

Gradient descent

Training neural networks

13

■ In general, neural networks are learned by gradient descent, using minibatches:

where

■ is the learning rate at update t

θ(z!y)

k

θ(z!y)

k

⌘(t)rθ(z→y)

k

`(i),

I ⌘(t)

Gradient descent

Training neural networks

13

■ In general, neural networks are learned by gradient descent, using minibatches:

where

■ is the learning rate at update t ■ is the loss on instance (minibatch) i

θ(z!y)

k

θ(z!y)

k

⌘(t)rθ(z→y)

k

`(i),

I ⌘(t)

I `(i)

Gradient descent

Training neural networks

13

■ In general, neural networks are learned by gradient descent, using minibatches:

where

■ is the learning rate at update t ■ is the loss on instance (minibatch) i ■ is the gradient of the loss with respect to the output weights

θ(z!y)

k

θ(z!y)

k

⌘(t)rθ(z→y)

k

`(i),

I ⌘(t)

I `(i)

I rθ(z→y)

k

`(i)

ts θ(z!y)

k

,

rθ(z→y)

k

`(i) =   @`(i) @✓(z!y)

k,1

, @`(i) @✓(z!y)

k,2

, . . . , @`(i) @✓(z!y)

k,Ky

 

Gradient descent

Training neural networks

14

Backpropagation

Training neural networks

14

■ If we don’t observe z, how can we learn the weights ?

Backpropagation

ts Θ(x!z)

Training neural networks

14

■ If we don’t observe z, how can we learn the weights ? ■ Backpropagation: compute a loss on y, and apply the chain rule from calculus to

compute a gradient on all parameters.

Backpropagation

ts Θ(x!z)

Training neural networks

14

■ If we don’t observe z, how can we learn the weights ? ■ Backpropagation: compute a loss on y, and apply the chain rule from calculus to

compute a gradient on all parameters.

Backpropagation

ts Θ(x!z)

■ Backpropagation as an algorithm: construct a directed acyclic computation graph

with nodes for inputs, outputs, hidden layers, parameters.

Training neural networks

15

■ Backpropagation as an algorithm: construct a directed acyclic computation graph

with nodes for inputs, outputs, hidden layers, parameters.

Backpropagation

x(i) z ˆ y `(i) y(i) Θ vx vz vˆ

y

vΘ gˆ

y

g gz gz vy vΘ g gˆ

y

Training neural networks

15

■ Backpropagation as an algorithm: construct a directed acyclic computation graph

with nodes for inputs, outputs, hidden layers, parameters.

Backpropagation

x(i) z ˆ y `(i) y(i) Θ vx vz vˆ

y

vΘ gˆ

y

g gz gz vy vΘ g gˆ

y

■ Forward pass: values (e.g. vx) go from parents to children

Training neural networks

15

■ Backpropagation as an algorithm: construct a directed acyclic computation graph

with nodes for inputs, outputs, hidden layers, parameters.

Backpropagation

x(i) z ˆ y `(i) y(i) Θ vx vz vˆ

y

vΘ gˆ

y

g gz gz vy vΘ g gˆ

y

■ Forward pass: values (e.g. vx) go from parents to children ■ Backward pass: gradients (e.g. gz) go from children to parents, implementing

the chain rule

Training neural networks

15

■ Backpropagation as an algorithm: construct a directed acyclic computation graph

with nodes for inputs, outputs, hidden layers, parameters.

Backpropagation

x(i) z ˆ y `(i) y(i) Θ vx vz vˆ

y

vΘ gˆ

y

g gz gz vy vΘ g gˆ

y

■ Forward pass: values (e.g. vx) go from parents to children ■ Backward pass: gradients (e.g. gz) go from children to parents, implementing

the chain rule

■ As long as the gradient is implemented for a layer/operation, you can add it to

the graph, and let automatic differentiation compute updates for every layer.

How to represent text for classification?

Another choice of R: word embeddings

16

■ Text is naturally viewed as a sequence of tokens w1, w2, …, wT

How to represent text for classification?

Another choice of R: word embeddings

16

■ Text is naturally viewed as a sequence of tokens w1, w2, …, wT ■ Context is lost when this sequence is converted to a bag-of-words.

How to represent text for classification?

Another choice of R: word embeddings

16

■ Text is naturally viewed as a sequence of tokens w1, w2, …, wT ■ Context is lost when this sequence is converted to a bag-of-words. ■ Instead, a lookup layer can compute embeddings (real-valued vectors) for

each type, resulting in a matrix

How to represent text for classification?

Another choice of R: word embeddings

16

w

=

• 1.36

1.77 0.71

• 0.25

0.11

• 1.36

0.03 0.71

• 0.45
• 0.23
• 0.58

1.43

• 1.27
• 0.71
• 0.23

0.69 1.43 1.88 0.84

• 0.33

0.11 1.36

• 1.08

0.84 0.14 0.11

• 0.18

… … … … … … … … …

• 0.067

0.93

• 5.6

0.74

• 0.07
• 0.067
• 0.36
• 5.6
• 1.58

t h e w e r e t h e w e r e b l a n d t a c

• s

b u t s t r

• n

g d r i n k s

n, resulting in a m where X(0) ∈ RKe×M.

Evaluating classifiers

17

Evaluating your classifier

18

Evaluating your classifier

■ Want to predict future performance, on unseen data.

18

Evaluating your classifier

■ Want to predict future performance, on unseen data. ■ It’s hard to predict the future. Do not evaluate on data that was already used for:

18

Evaluating your classifier

■ Want to predict future performance, on unseen data. ■ It’s hard to predict the future. Do not evaluate on data that was already used for: ■ training

18

Evaluating your classifier

■ Want to predict future performance, on unseen data. ■ It’s hard to predict the future. Do not evaluate on data that was already used for: ■ training ■ hyperparameter selection

18

Evaluating your classifier

■ Want to predict future performance, on unseen data. ■ It’s hard to predict the future. Do not evaluate on data that was already used for: ■ training ■ hyperparameter selection ■ selecting classification model, model structure

18

Evaluating your classifier

■ Want to predict future performance, on unseen data. ■ It’s hard to predict the future. Do not evaluate on data that was already used for: ■ training ■ hyperparameter selection ■ selecting classification model, model structure ■ preprocessing decisions, such as vocabulary selection

18

Evaluating your classifier

■ Want to predict future performance, on unseen data. ■ It’s hard to predict the future. Do not evaluate on data that was already used for: ■ training ■ hyperparameter selection ■ selecting classification model, model structure ■ preprocessing decisions, such as vocabulary selection ■ Even if you follow all these rules, you will probably still over-estimate your

classifier’s performance, because real future data will differ from your test set in ways that you cannot anticipate.

18

Accuracy

19

Accuracy

■ Most basic metric: accuracy. How often is the classifier right?

The problem with accuracy is rare labels, also known as class imbalance.

19

acc(y, ˆ y) = 1 N

N

X

i=1

δ(y(i) = ˆ y).

Accuracy

■ Most basic metric: accuracy. How often is the classifier right?

The problem with accuracy is rare labels, also known as class imbalance.

19

acc(y, ˆ y) = 1 N

N

X

i=1

δ(y(i) = ˆ y).

Accuracy

■ Most basic metric: accuracy. How often is the classifier right?

The problem with accuracy is rare labels, also known as class imbalance.

■ Consider a system for detecting whether a tweet is written in Telugu.

19

acc(y, ˆ y) = 1 N

N

X

i=1

δ(y(i) = ˆ y).

Accuracy

■ Most basic metric: accuracy. How often is the classifier right?

The problem with accuracy is rare labels, also known as class imbalance.

■ Consider a system for detecting whether a tweet is written in Telugu. ■ 0.3% of tweets are written in Telugu [Bergsma et al. 2012].

19

acc(y, ˆ y) = 1 N

N

X

i=1

δ(y(i) = ˆ y).

Accuracy

■ Most basic metric: accuracy. How often is the classifier right?

The problem with accuracy is rare labels, also known as class imbalance.

■ Consider a system for detecting whether a tweet is written in Telugu. ■ 0.3% of tweets are written in Telugu [Bergsma et al. 2012]. ■ A system that says ŷ = NotTelugu 100% of the time is 99.7% accurate.

19

acc(y, ˆ y) = 1 N

N

X

i=1

δ(y(i) = ˆ y).

Beyond “right” and “wrong”

20

correct labels predicted labels

Beyond “right” and “wrong”

■ For any label, there are two ways to be “wrong:”

20

correct labels predicted labels

Beyond “right” and “wrong”

■ For any label, there are two ways to be “wrong:” ■ False positive: the system incorrectly

predicts the label.

20

correct labels predicted labels

Beyond “right” and “wrong”

■ For any label, there are two ways to be “wrong:” ■ False positive: the system incorrectly

predicts the label.

■ False negative: the system incorrectly fails

to predict the label.

20

correct labels predicted labels

Beyond “right” and “wrong”

■ For any label, there are two ways to be “wrong:” ■ False positive: the system incorrectly

predicts the label.

■ False negative: the system incorrectly fails

to predict the label.

■ Similarly, there are two ways to be “right:”

20

correct labels predicted labels

Beyond “right” and “wrong”

■ For any label, there are two ways to be “wrong:” ■ False positive: the system incorrectly

predicts the label.

■ False negative: the system incorrectly fails

to predict the label.

■ Similarly, there are two ways to be “right:” ■ True positive: the system correctly predicts

the label.

20

correct labels predicted labels

Beyond “right” and “wrong”

■ For any label, there are two ways to be “wrong:” ■ False positive: the system incorrectly

predicts the label.

■ False negative: the system incorrectly fails

to predict the label.

■ Similarly, there are two ways to be “right:” ■ True positive: the system correctly predicts

the label.

■ True negative: the system correctly predicts

that the label does not apply to this instance.

20

correct labels predicted labels

Precision and recall

21

correct labels predicted labels

Precision and recall

21

correct labels predicted labels

recall = TP TP + FN =

■ Recall: fraction of positive instances that were

correctly classified.

Precision and recall

21

correct labels predicted labels

recall = TP TP + FN =

■ Recall: fraction of positive instances that were

correctly classified.

■ The “never Telugu” classifier has 0 recall.

Precision and recall

21

correct labels predicted labels

recall = TP TP + FN =

■ Recall: fraction of positive instances that were

correctly classified.

■ The “never Telugu” classifier has 0 recall. ■ The “always Telugu” classifier has perfect recall.

Precision and recall

21

correct labels predicted labels

recall = TP TP + FN =

precision = TP TP + FP =

■ Recall: fraction of positive instances that were

correctly classified.

■ The “never Telugu” classifier has 0 recall. ■ The “always Telugu” classifier has perfect recall.

■ Precision: fraction of positive predictions that were

correct.

Precision and recall

21

correct labels predicted labels

recall = TP TP + FN =

precision = TP TP + FP =

■ Recall: fraction of positive instances that were

correctly classified.

■ The “never Telugu” classifier has 0 recall. ■ The “always Telugu” classifier has perfect recall.

■ Precision: fraction of positive predictions that were

correct.

■ The “never Telugu” classifier 0 precision.

Precision and recall

21

correct labels predicted labels

recall = TP TP + FN =

precision = TP TP + FP =

■ Recall: fraction of positive instances that were

correctly classified.

■ The “never Telugu” classifier has 0 recall. ■ The “always Telugu” classifier has perfect recall.

■ Precision: fraction of positive predictions that were

correct.

■ The “never Telugu” classifier 0 precision. ■ The “always Telugu” classifier has 0.003 precision.

Combining precision and recall

22

■ Inherent tradeoff between precision and recall. Choice is problem-specific.

Combining precision and recall

22

■ Inherent tradeoff between precision and recall. Choice is problem-specific. ■ For a preliminary medical diagnosis, we might prefer high recall. False positives

can be screened out later.

Combining precision and recall

22

■ Inherent tradeoff between precision and recall. Choice is problem-specific. ■ For a preliminary medical diagnosis, we might prefer high recall. False positives

can be screened out later.

■ The “beyond reasonable doubt” standard of U.S. criminal law implies a

preference for high precision.

Combining precision and recall

22

■ Inherent tradeoff between precision and recall. Choice is problem-specific. ■ For a preliminary medical diagnosis, we might prefer high recall. False positives

can be screened out later.

■ The “beyond reasonable doubt” standard of U.S. criminal law implies a

preference for high precision.

■ Most often, we weight them equally using F1 measure: harmonic mean of precision

and recall.

Combining precision and recall

22

F1 = 2 · precision · recall precision + recall

■ Inherent tradeoff between precision and recall. Choice is problem-specific. ■ For a preliminary medical diagnosis, we might prefer high recall. False positives

can be screened out later.

■ The “beyond reasonable doubt” standard of U.S. criminal law implies a

preference for high precision.

■ Most often, we weight them equally using F1 measure: harmonic mean of precision

and recall.

Combining precision and recall

22

F1 = 2 · precision · recall precision + recall

min(precision, recall) ≤ F1 ≤ 2 · min(precision, recall)

■ Inherent tradeoff between precision and recall. Choice is problem-specific. ■ For a preliminary medical diagnosis, we might prefer high recall. False positives

can be screened out later.

■ The “beyond reasonable doubt” standard of U.S. criminal law implies a

preference for high precision.

■ Most often, we weight them equally using F1 measure: harmonic mean of precision

and recall.

■ Can generalize F-measure to adjust the tradeoff, such that recall is β-times as

important as precision:

Combining precision and recall

22

F1 = 2 · precision · recall precision + recall

Fβ = (1 + β2) precision · recall (β2 · precision) + recall

min(precision, recall) ≤ F1 ≤ 2 · min(precision, recall)

Trading off precision and recall: ROC curve

23

Trading off precision and recall: ROC curve

23

Evaluating multi-class classification

24

■ Recall and precision imply binary classification: each instance is either positive or

negative.

Evaluating multi-class classification

24

■ Recall and precision imply binary classification: each instance is either positive or

negative.

■ In multi-class classification, each instance is positive for one class, and negative for

all other classes.

Evaluating multi-class classification

24

■ Recall and precision imply binary classification: each instance is either positive or

negative.

■ In multi-class classification, each instance is positive for one class, and negative for

all other classes.

■ Two ways to combine performance across classes:

Evaluating multi-class classification

24

■ Recall and precision imply binary classification: each instance is either positive or

negative.

■ In multi-class classification, each instance is positive for one class, and negative for

all other classes.

■ Two ways to combine performance across classes: ■ Macro F1: Compute F1 per class, then average across all classes.

Evaluating multi-class classification

24

■ Recall and precision imply binary classification: each instance is either positive or

negative.

■ In multi-class classification, each instance is positive for one class, and negative for

all other classes.

■ Two ways to combine performance across classes: ■ Macro F1: Compute F1 per class, then average across all classes. ■ Micro F1: Compute total number of true positives, false positives, false

negatives pooled across all classes, and compute a single F1 .

Evaluating multi-class classification

24

■ Recall and precision imply binary classification: each instance is either positive or

negative.

■ In multi-class classification, each instance is positive for one class, and negative for

all other classes.

■ Two ways to combine performance across classes: ■ Macro F1: Compute F1 per class, then average across all classes. ■ Micro F1: Compute total number of true positives, false positives, false

negatives pooled across all classes, and compute a single F1 .

Evaluating multi-class classification

24

→ Weights all classes equally, regardless of frequency.

■ Recall and precision imply binary classification: each instance is either positive or

negative.

■ In multi-class classification, each instance is positive for one class, and negative for

all other classes.

■ Two ways to combine performance across classes: ■ Macro F1: Compute F1 per class, then average across all classes. ■ Micro F1: Compute total number of true positives, false positives, false

negatives pooled across all classes, and compute a single F1 .

Evaluating multi-class classification

24

→ Weights all classes equally, regardless of frequency. → Emphasizes performance on high frequency classes.

Comparing classifiers

25

■ Suppose two teams build classifiers to solve a

problem:

Comparing classifiers

25

■ Suppose two teams build classifiers to solve a

problem:

■ C1 gets 82% accuracy

Comparing classifiers

25

■ Suppose two teams build classifiers to solve a

problem:

■ C1 gets 82% accuracy ■ C2 gets 73% accuracy

Comparing classifiers

25

■ Suppose two teams build classifiers to solve a

problem:

■ C1 gets 82% accuracy ■ C2 gets 73% accuracy ■ Will C1 be more accurate in the future?

Comparing classifiers

25

■ Suppose two teams build classifiers to solve a

problem:

■ C1 gets 82% accuracy ■ C2 gets 73% accuracy ■ Will C1 be more accurate in the future? ■ What if the test set has 1000 examples?

Comparing classifiers

25

■ Suppose two teams build classifiers to solve a

problem:

■ C1 gets 82% accuracy ■ C2 gets 73% accuracy ■ Will C1 be more accurate in the future? ■ What if the test set has 1000 examples? ■ What if the test set has 11 examples?

Comparing classifiers

25

Hypothesis testing

26

■ Consider two hypotheses that explain the observed data:

Hypothesis testing

26

■ Consider two hypotheses that explain the observed data: ■ H1: C1 is more accurate than C2 and therefore can be expected to be more

accurate in the future (in the limit of an infinite number of independent evaluations).

Hypothesis testing

26

■ Consider two hypotheses that explain the observed data: ■ H1: C1 is more accurate than C2 and therefore can be expected to be more

accurate in the future (in the limit of an infinite number of independent evaluations).

■ H0: C1 is not more accurate than C2 , and its superior performance on the test

set was due only to luck. This is the null hypothesis.

Hypothesis testing

26

■ Consider two hypotheses that explain the observed data: ■ H1: C1 is more accurate than C2 and therefore can be expected to be more

accurate in the future (in the limit of an infinite number of independent evaluations).

■ H0: C1 is not more accurate than C2 , and its superior performance on the test

set was due only to luck. This is the null hypothesis.

■ If the test set is small, H0 might be true.

Hypothesis testing

26

■ Consider two hypotheses that explain the observed data: ■ H1: C1 is more accurate than C2 and therefore can be expected to be more

accurate in the future (in the limit of an infinite number of independent evaluations).

■ H0: C1 is not more accurate than C2 , and its superior performance on the test

set was due only to luck. This is the null hypothesis.

■ If the test set is small, H0 might be true. ■ If the test set is large, probability of observing a 9% difference in accuracy

(73% → 82%) becomes vanishingly small unless C1 really is more accurate.

Hypothesis testing

26

■ Consider two hypotheses that explain the observed data: ■ H1: C1 is more accurate than C2 and therefore can be expected to be more

accurate in the future (in the limit of an infinite number of independent evaluations).

■ H0: C1 is not more accurate than C2 , and its superior performance on the test

set was due only to luck. This is the null hypothesis.

■ If the test set is small, H0 might be true. ■ If the test set is large, probability of observing a 9% difference in accuracy

(73% → 82%) becomes vanishingly small unless C1 really is more accurate.

■ These probabilities are quantified by hypothesis testing.

Hypothesis testing

26

The binomial test

27

■ If the two classifiers are equally accurate, then each time they disagree, they are

equally likely to be correct.

The binomial test

27

■ If the two classifiers are equally accurate, then each time they disagree, they are

equally likely to be correct.

■ Over 30 such disagreements, each classifier will “win” roughly half the time.

The binomial test

27

■ If the two classifiers are equally accurate, then each time they disagree, they are

equally likely to be correct.

■ Over 30 such disagreements, each classifier will “win” roughly half the time. ■ The total probability mass in the pink region is less than 5%. If the data are in this

region, we reject the null hypothesis with p < 0.05.

The binomial test

27

Other hypothesis tests

28

■ The binomial test compares two classifiers in terms of accuracy. It can be computed

in closed form using e.g. SciPy or R.

Other hypothesis tests

28

■ The binomial test compares two classifiers in terms of accuracy. It can be computed

in closed form using e.g. SciPy or R.

■ Hypotheses about other metrics, such as F1, cannot be tested in this way.

Other hypothesis tests

28

■ The binomial test compares two classifiers in terms of accuracy. It can be computed

in closed form using e.g. SciPy or R.

■ Hypotheses about other metrics, such as F1, cannot be tested in this way. ■ For these hypotheses, best approach is randomization: randomly sample many test

sets, and count how often each hypothesis holds.

Other hypothesis tests

28

Comparing classifiers

29

F1

Comparing classifiers

29

Nils Reimers and Iryna Gurevych. Reporting score distributions makes a difference: Performance study of LSTM-networks for sequence tagging. EMNLP 2017.

F1

Two recent, state-of-the-art systems for NER are

proposed by Ma and Hovy (2016)5 and by Lample et al. (2016)6. Lample et al. report an F1-score of 90.94% and Ma and Hovy report an F1-score of 91.21%. Ma and Hovy draw the conclusion that

their system achieves a significant improvement

• ver the system by Lample et al.

Comparing classifiers

29

Nils Reimers and Iryna Gurevych. Reporting score distributions makes a difference: Performance study of LSTM-networks for sequence tagging. EMNLP 2017.

F1

Two recent, state-of-the-art systems for NER are

proposed by Ma and Hovy (2016)5 and by Lample et al. (2016)6. Lample et al. report an F1-score of 90.94% and Ma and Hovy report an F1-score of 91.21%. Ma and Hovy draw the conclusion that

their system achieves a significant improvement

• ver the system by Lample et al.

Lample et al. (reported)

Comparing classifiers

29

Nils Reimers and Iryna Gurevych. Reporting score distributions makes a difference: Performance study of LSTM-networks for sequence tagging. EMNLP 2017.

F1

Two recent, state-of-the-art systems for NER are

proposed by Ma and Hovy (2016)5 and by Lample et al. (2016)6. Lample et al. report an F1-score of 90.94% and Ma and Hovy report an F1-score of 91.21%. Ma and Hovy draw the conclusion that

their system achieves a significant improvement

• ver the system by Lample et al.

Ma & Hovy (reported) Lample et al. (reported)

Announcements

30

■ Project 1: Text classification will be available after class today,

due Friday September 25

■ Han will lead recitation this Friday to introduce Project 1, and cover environment

setup (Python, Jupyter, NumPy, PyTorch).