Introduction to Machine Learning: Classification and The Noisy - - PowerPoint PPT Presentation

Introduction to Machine Learning: Classification and The Noisy Channel Model

CMSC 473/673 UMBC

Some slides adapted from 3SLP

Outline

Classification Why incorporate uncertainty Classification with Bayes Rule Example: Email Classifier Evaluation

Probabilistic Classification

𝑞 𝑍 𝑌) ∝ 𝑞 𝑌 𝑍) ∗ 𝑞(𝑍)

𝑞 𝑍 𝑌) = ℎ(𝑌; 𝑍)

Discriminatively trained classifier Generatively trained classifier

Directly model the posterior Model the posterior with Bayes rule

Outline

Classification Why incorporate uncertainty Classification with Bayes Rule Example: Email Classifier Evaluation

Classification

POLITICS TERRORISM SPORTS TECH HEALTH FINANCE …

Three people have been fatally shot, and five people, including a mayor, were seriously wounded as a result of a Shining Path attack today against a community in Junin department, central Peruvian mountain region.

Classification

POLITICS TERRORISM SPORTS TECH HEALTH FINANCE …

Three people have been fatally shot, and five people, including a mayor, were seriously wounded as a result of a Shining Path attack today against a community in Junin department, central Peruvian mountain region.

Classification

POLITICS TERRORISM SPORTS TECH HEALTH FINANCE …

Electronic alerts have been used to assist the authorities in moments of chaos and potential danger: after the Boston bombing in 2013, when the Boston suspects were still at large, and last month in Los Angeles, during an active shooter scare at the airport.

Source: http://www.nytimes.com/2016/09/20/nyregion/cellphone-alerts-used-in-search-of- manhattan-bombing-suspect.html

Classification

POLITICS TERRORISM SPORTS TECH HEALTH FINANCE …

Electronic alerts have been used to assist the authorities in moments of chaos and potential danger: after the Boston bombing in 2013, when the Boston suspects were still at large, and last month in Los Angeles, during an active shooter scare at the airport.

Source: http://www.nytimes.com/2016/09/20/nyregion/cellphone-alerts-used-in-search-of- manhattan-bombing-suspect.html

Classify with Uncertainty

Use probabilities

Classify with Uncertainty

Use probabilities*

*There are non-probabilistic ways to handle uncertainty… but probabilities sure are handy!

Classification

POLITICS .05 TERRORISM .48 SPORTS .0001 TECH .39 HEALTH .0001 FINANCE .0002 …

Electronic alerts have been used to assist the authorities in moments of chaos and potential danger: after the Boston bombing in 2013, when the Boston suspects were still at large, and last month in Los Angeles, during an active shooter scare at the airport.

Source: http://www.nytimes.com/2016/09/20/nyregion/cellphone-alerts-used-in-search-of- manhattan-bombing-suspect.html

Text Classification

Assigning subject categories, topics, or genres Spam detection Authorship identification Age/gender identification Language Identification Sentiment analysis …

Text Classification

Assigning subject categories, topics, or genres Spam detection Authorship identification Age/gender identification Language Identification Sentiment analysis …

Input:

a document a fixed set of classes C = {c1, c2,…, cJ}

Output: a predicted class c from C

Text Classification

Assigning subject categories, topics, or genres Spam detection Authorship identification Age/gender identification Language Identification Sentiment analysis …

Input:

a document linguistic blob a fixed set of classes C = {c1, c2,…, cJ}

Output: a predicted class c from C

Text Classification: Hand-coded Rules?

Assigning subject categories, topics, or genres Spam detection Authorship identification Age/gender identification Language Identification Sentiment analysis …

Rules based on combinations of words or other features

spam: black-list-address OR (“dollars” AND “have been selected”)

Accuracy can be high

If rules carefully refined by expert

Building and maintaining these rules is expensive Can humans faithfully assign uncertainty?

Text Classification: Supervised Machine Learning

Assigning subject categories, topics, or genres Spam detection Authorship identification Age/gender identification Language Identification Sentiment analysis …

Input:

a document d a fixed set of classes C = {c1, c2,…, cJ} A training set of m hand-labeled documents (d1,c1),....,(dm,cm)

Output:

a learned classifier γ that maps documents to classes

Text Classification: Supervised Machine Learning

Assigning subject categories, topics, or genres Spam detection Authorship identification Age/gender identification Language Identification Sentiment analysis …

Input:

a document d a fixed set of classes C = {c1, c2,…, cJ} A training set of m hand-labeled documents (d1,c1),....,(dm,cm)

Output:

a learned classifier γ that maps documents to classes

Naïve Bayes Logistic regression Support-vector machines k-Nearest Neighbors …

Text Classification: Supervised Machine Learning

Assigning subject categories, topics, or genres Spam detection Authorship identification Age/gender identification Language Identification Sentiment analysis …

Naïve Bayes Logistic regression Support-vector machines k-Nearest Neighbors …

Input:

a document d a fixed set of classes C = {c1, c2,…, cJ} A training set of m hand-labeled documents (d1,c1),....,(dm,cm)

Output:

a learned classifier γ that maps documents to classes

Multi-class Classification

Given input 𝑦, predict discrete label 𝑧

Multi-label Classification

Multi-class Classification

Given input 𝑦, predict discrete label 𝑧

If 𝑧 ∈ {0,1} (or 𝑧 ∈ {True, False}), then a binary classification task

Multi-label Classification

Multi-class Classification

Given input 𝑦, predict discrete label 𝑧

If 𝑧 ∈ {0,1} (or 𝑧 ∈ {True, False}), then a binary classification task If 𝑧 ∈ {0,1, … , 𝐿 − 1} (for finite K), then a multi-class classification task Q: What are some examples

• f multi-class classification?

Multi-label Classification

Multi-class Classification

Given input 𝑦, predict discrete label 𝑧

If 𝑧 ∈ {0,1} (or 𝑧 ∈ {True, False}), then a binary classification task If 𝑧 ∈ {0,1, … , 𝐿 − 1} (for finite K), then a multi-class classification task

Multi-label Classification

Single

• utput

Multi-

• utput

If multiple 𝑧𝑚 are predicted, then a multi- label classification task

Multi-class Classification

Given input 𝑦, predict discrete label 𝑧

If 𝑧 ∈ {0,1} (or 𝑧 ∈ {True, False}), then a binary classification task If 𝑧 ∈ {0,1, … , 𝐿 − 1} (for finite K), then a multi-class classification task

Multi-label Classification

Single

• utput

Multi-

• utput

Given input 𝑦, predict multiple discrete labels 𝑧 = (𝑧1, … , 𝑧𝑀)

If multiple 𝑧𝑚 are predicted, then a multi- label classification task

Multi-class Classification

Given input 𝑦, predict discrete label 𝑧

If 𝑧 ∈ {0,1} (or 𝑧 ∈ {True, False}), then a binary classification task If 𝑧 ∈ {0,1, … , 𝐿 − 1} (for finite K), then a multi-class classification task

Multi-label Classification

Single

• utput

Multi-

• utput

Given input 𝑦, predict multiple discrete labels 𝑧 = (𝑧1, … , 𝑧𝑀)

If multiple 𝑧𝑚 are predicted, then a multi- label classification task Each 𝑧𝑚 could be binary or multi-class

Outline

Classification Why incorporate uncertainty Classification with Bayes Rule Example: Email Classifier Evaluation

Probabilistic Text Classification

Assigning subject categories, topics, or genres Spam detection Authorship identification Age/gender identification Language Identification Sentiment analysis …

𝑞 𝑍 𝑌) = 𝑞 𝑌 𝑍) ∗ 𝑞(𝑍) 𝑞(𝑌)

class

• bserved

data

Probabilistic Text Classification

Assigning subject categories, topics, or genres Spam detection Authorship identification Age/gender identification Language Identification Sentiment analysis …

class

• bserved

data prior probability of class

• bservation likelihood (averaged over all classes)

class-based likelihood (language model)

𝑞 𝑍 𝑌) = 𝑞 𝑌 𝑍) ∗ 𝑞(𝑍) 𝑞(𝑌)

Probabilistic Text Classification

Assigning subject categories, topics, or genres Spam detection Authorship identification Age/gender identification Language Identification Sentiment analysis …

class

• bserved

data class-based likelihood (language model) prior probability of class

• bservation likelihood (averaged over all classes)

𝑞 𝑍 𝑌) = 𝑞 𝑌 𝑍) ∗ 𝑞(𝑍) 𝑞(𝑌)

Classification with Bayes Rule argmax𝑍𝑞 𝑍 𝑌)

Classification with Bayes Rule argmax𝑍 𝑞 𝑌 𝑍) ∗ 𝑞(𝑍) 𝑞(𝑌)

Classification with Bayes Rule argmax𝑍 𝑞 𝑌 𝑍) ∗ 𝑞(𝑍) 𝑞(𝑌)

constant with respect to Y

Classification with Bayes Rule argmax𝑍𝑞 𝑌 𝑍) ∗ 𝑞(𝑍)

Classification with Bayes Rule argmax𝑍 log 𝑞 𝑌 𝑍) + log 𝑞(𝑍)

Classification (labels) with Bayes Rule

argmax𝑍 log 𝑞 𝑌 𝑍) + log 𝑞(𝑍)

how well does text X represent label Y? how likely is label Y overall?

For “simple” or “flat” labels: * iterate through labels * evaluate score for each label, keeping only the best (n best) * return the best (or n best) label and score

Classification/Decoding with Bayes Rule

argmax𝑍 log 𝑞 𝑌 𝑍) + log 𝑞(𝑍)

how well does text (complex input) X represent text (complex output) Y? how likely is text (complex

• utput) Y overall?

* iterate through labels * evaluate score for each label, keeping only the best (n best) * return the best (or n best) label and score

If Y is a string (or some complex structure), this can be complicated

Terminology: Noisy Channel Model

For us, it means using Bayes rule to build a probabilistic classifier/system

Image source: https://encrypted-tbn0.gstatic.com/images?q=tbn:ANd9GcSZZUgncH-1nsicmUNzhOT2ttW8DZBEQS1HmK5JwqCET0X1gnX5qw

Noisy Channel Model

Noisy Channel Model

what I want to tell you “sports”

Noisy Channel Model

what I want to tell you “sports” what you actually see “The Os lost again…”

Noisy Channel Model

what I want to tell you “sports” what you actually see “The Os lost again…” Decode hypothesized intent “sad stories” “sports”

Noisy Channel Model

what I want to tell you “sports” what you actually see “The Os lost again…” Decode Rerank hypothesized intent “sad stories” “sports” reweight according to what’s likely “sports”

Noisy Channel

Machine translation Speech-to-text Spelling correction Text normalization Part-of-speech tagging Morphological analysis Image captioning …

𝑞 𝑍 𝑌) = 𝑞 𝑌 𝑍) ∗ 𝑞(𝑍) 𝑞(𝑌)

possible (clean)

• utput
• bserved

(noisy) text translation/ decode model (clean) language model

• bservation (noisy) likelihood

Noisy Channel

Machine translation Speech-to-text Spelling correction Text normalization Part-of-speech tagging Morphological analysis Image captioning …

possible (clean)

• utput
• bserved

(noisy) text (clean) language model

• bservation (noisy) likelihood

translation/ decode model

𝑞 𝑍 𝑌) = 𝑞 𝑌 𝑍) ∗ 𝑞(𝑍) 𝑞(𝑌)

Language Model

Use any of the language modeling algorithms we’ve learned Unigram, bigram, trigram Add-λ, interpolation, backoff (Later: Maxent, RNNs, hierarchical Bayesian LMs, …)

Probabilistic Classification

𝑞 𝑍 𝑌) ∝ 𝑞 𝑌 𝑍) ∗ 𝑞(𝑍)

𝑞 𝑍 𝑌) = ℎ(𝑌; 𝑍)

Discriminatively trained classifier Generatively trained classifier

Directly model the posterior Model the posterior with Bayes rule

Noisy Channel Model Decoding Discriminative training (e.g., maxent models: we’ll cover these soon)

Outline

Classification Why incorporate uncertainty Classification with Bayes Rule Example: Email Classifier Classification

Problem: Develop a Probabilistic Email Classifier

Input: an email (all text) Output (Google categories): Primary, Social, Forums, Spam Approach #1: Using Bayes rule Approach #2 (later classes): Discriminatively trained

Problem: Develop a Probabilistic Email Classifier

Input: an email (all text) Output (Google categories): Primary, Social, Forums, Spam Approach #1: Using Bayes rule Approach #2 (later classes): Discriminatively trained

Q: What type of classification problem is this?

Problem: Develop a Probabilistic Email Classifier

Input: an email (all text) Output (Google categories): Primary, Social, Forums, Spam Approach #1: Using Bayes rule Approach #2 (later classes): Discriminatively trained

Q: What type of classification problem is this? A: multi-class (single label) classifier

Classify Using Bayes Rule 𝑞 𝑍 𝑌) ∝ 𝑞 𝑌 𝑍) ∗ 𝑞(𝑍)

Classify Using Bayes Rule 𝑞 𝑍 𝑌) ∝ 𝑞 𝑌 𝑍) ∗ 𝑞(𝑍)

Q: Why is p(Y | X) what we want to model?

Classify Using Bayes Rule 𝑞 𝑍 𝑌) ∝ 𝑞 𝑌 𝑍) ∗ 𝑞(𝑍)

Q: Why is p(Y | X) what we want to model? Q: To classify a document, do we need to find the normalizing constant?

Classify Using Bayes Rule 𝑞 𝑍 𝑌) ∝ 𝑞 𝑌 𝑍) ∗ 𝑞(𝑍)

Q: Why is p(Y | X) what we want to model? Q: To classify a document, do we need to find the normalizing constant? Q: If we can compute p(Y | X), up to a constant, how do we find the predicted label?

Classify Using Bayes Rule 𝑞 𝑍 𝑌) ∝ 𝑞 𝑌 𝑍) ∗ 𝑞(𝑍)

Won’t you please donate?

𝑞 ) ∝ 𝑞 )𝑞( )

Primary Primary Primary

Won’t you please donate?

A Closer Look at 𝑞 )

This is a class specific language model

Primary

Won’t you please donate?

A Closer Look at 𝑞 )

This is a class specific language model 𝑞 ) is different from 𝑞 ) is different from 𝑞 ) …

Primary

Won’t you please donate?

Primary

Won’t you please donate?

Social

Won’t you please donate?

Forums

Won’t you please donate?

A Closer Look at 𝑞 )

This is a class specific language model To learn 𝑞 ): For each class Class:

Get a bunch of Class documents 𝐸Class Learn a new language model 𝑞Class on just 𝐸Class

Primary

Won’t you please donate?

Class

Won’t you please donate?

Two Ways to Learn Class-specific Count-based Language Models

• 1. Create different count

table(s) 𝑑Class(… ) for each Class

e.g., record separate trigram counts for Primary vs. Social

• vs. Forums vs. Spam

Two Ways to Learn Class-specific Count-based Language Models

• 1. Create different count table(s)

𝑑Class(… ) for each Class

e.g., record separate trigram counts for Primary vs. Social vs. Forums vs. Spam

OR

• 2. Add a dimension to your existing

tables 𝑑(Class, … )

e.g., record how often each trigram

• ccurs within Primary vs. Social vs.

Forums vs. Spam documents

Two Ways to Learn Class-specific Count-based Language Models

• 1. Create different count table(s)

𝑑Class(… ) for each Class

e.g., record separate trigram counts for Primary vs. Social vs. Forums vs. Spam

OR

• 2. Add a dimension to your existing

tables 𝑑(Class, … )

e.g., record how often each trigram

• ccurs within Primary vs. Social vs.

Forums vs. Spam documents

Q: Are these two conceptually the same?

Two Ways to Learn Class-specific Count-based Language Models

• 1. Create different count table(s)

𝑑Class(… ) for each Class

e.g., record separate trigram counts for Primary vs. Social vs. Forums vs. Spam

OR

• 2. Add a dimension to your existing

tables 𝑑(Class, … )

e.g., record how often each trigram

• ccurs within Primary vs. Social vs.

Forums vs. Spam documents

Q: Are these two conceptually the same? Q: How might the option you choose influence implementation (or vice versa)?

Two Ways to Learn Class-specific Count-based Language Models

• 1. Create different count table(s)

𝑑Class(… ) for each Class

e.g., record separate trigram counts for Primary vs. Social vs. Forums vs. Spam

OR

• 2. Add a dimension to your existing

tables 𝑑(Class, … )

e.g., record how often each trigram

• ccurs within Primary vs. Social vs.

Forums vs. Spam documents

Q: Are these two conceptually the same? Q: How might the option you choose influence implementation (or vice versa)? Q: Will one approach always be better than the

• ther?

A Closer Look at 𝑞( )

This is the prior probability of each class Answers the question: without knowing anything specific about a document, how likely is each class?

Primary

A Closer Look at 𝑞( )

This is the prior probability of each class Answers the question: without knowing anything specific about a document, how likely is each class?

Primary

Q: What’s an easy way to estimate it?

A Closer Look at 𝑞( )

This is the prior probability of each class Answers the question: without knowing anything specific about a document, how likely is each class?

Primary

Q: What’s an easy way to estimate it? Q: Could we use our smoothing techniques?

Outline

Classification Why incorporate uncertainty Classification with Bayes Rule Example: Email Classifier Evaluation

Experimenting with Machine Learning Models

All your data Training Data

Dev Data Test Data

Rule #1

Experimenting with Machine Learning Models

What is “correct?” What is working “well?”

Training Data

Dev Data Test Data

Learn model parameters from training set

set hyper- parameters

Experimenting with Machine Learning Models

What is “correct?” What is working “well?”

Training Data

Dev Data Test Data

set hyper- parameters

Evaluate the learned model on dev with that hyperparameter setting Learn model parameters from training set

Experimenting with Machine Learning Models

What is “correct?” What is working “well?”

Training Data

Dev Data Test Data

set hyper- parameters

Evaluate the learned model on dev with that hyperparameter setting Learn model parameters from training set perform final evaluation on test, using the hyperparameters that

• ptimized dev performance and

retraining the model

Experimenting with Machine Learning Models

What is “correct?” What is working “well?”

Training Data

Dev Data Test Data

set hyper- parameters

Evaluate the learned model on dev with that hyperparameter setting Learn model parameters from training set perform final evaluation on test, using the hyperparameters that

• ptimized dev performance and

retraining the model

Rule 1: DO NOT ITERATE ON THE TEST DATA

Evaluation: the 2-by-2 contingency table

Actually Correct Actually Incorrect Selected/ Guessed Not selected/ not guessed

Evaluation: the 2-by-2 contingency table

Actually Correct Actually Incorrect Selected/ Guessed Not selected/ not guessed

Classes/Choices

Evaluation: the 2-by-2 contingency table

Actually Correct Actually Incorrect Selected/ Guessed True Positive (TP) Not selected/ not guessed

Classes/Choices

Correct Guessed

Evaluation: the 2-by-2 contingency table

Actually Correct Actually Incorrect Selected/ Guessed True Positive (TP) False Positive (FP) Not selected/ not guessed

Classes/Choices

Correct Guessed Correct Guessed

Evaluation: the 2-by-2 contingency table

Actually Correct Actually Incorrect Selected/ Guessed True Positive (TP) False Positive (FP) Not selected/ not guessed False Negative (FN)

Classes/Choices

Correct Guessed Correct Guessed Correct Guessed

Evaluation: the 2-by-2 contingency table

Actually Correct Actually Incorrect Selected/ Guessed True Positive (TP) False Positive (FP) Not selected/ not guessed False Negative (FN) True Negative (TN)

Classes/Choices

Correct Guessed Correct Guessed Correct Guessed Correct Guessed

Accuracy, Precision, and Recall

Accuracy: % of items correct

Actually Correct Actually Incorrect Selected/Guessed True Positive (TP) False Positive (FP) Not select/not guessed False Negative (FN) True Negative (TN)

TP + TN TP + FP + FN + TN

Accuracy, Precision, and Recall

Accuracy: % of items correct Precision: % of selected items that are correct

Actually Correct Actually Incorrect Selected/Guessed True Positive (TP) False Positive (FP) Not select/not guessed False Negative (FN) True Negative (TN)

TP TP + FP TP + TN TP + FP + FN + TN

Accuracy, Precision, and Recall

Accuracy: % of items correct Precision: % of selected items that are correct Recall: % of correct items that are selected

Actually Correct Actually Incorrect Selected/Guessed True Positive (TP) False Positive (FP) Not select/not guessed False Negative (FN) True Negative (TN)

TP TP + FP TP TP + FN TP + TN TP + FP + FN + TN

A combined measure: F

Weighted (harmonic) average of Precision & Recall 𝐺 = 1 𝛽 1 𝑄 + (1 − 𝛽) 1 𝑆

A combined measure: F

Weighted (harmonic) average of Precision & Recall 𝐺 = 1 𝛽 1 𝑄 + (1 − 𝛽) 1 𝑆 = 1 + 𝛾2 ∗ 𝑄 ∗ 𝑆 (𝛾2 ∗ 𝑄) + 𝑆

algebra (not important)

A combined measure: F

Weighted (harmonic) average of Precision & Recall Balanced F1 measure: β=1 𝐺 = 1 + 𝛾2 ∗ 𝑄 ∗ 𝑆 (𝛾2 ∗ 𝑄) + 𝑆 𝐺

1 = 2 ∗ 𝑄 ∗ 𝑆

𝑄 + 𝑆

Micro- vs. Macro-Averaging

If we have more than one class, how do we combine multiple performance measures into one quantity? Macroaveraging: Compute performance for each class, then average. Microaveraging: Collect decisions for all classes, compute contingency table, evaluate.

• Sec. 15.2.4

Micro- vs. Macro-Averaging: Example

Truth : yes Truth : no Classifier: yes 10 10 Classifier: no 10 970 Truth : yes Truth : no Classifier: yes 90 10 Classifier: no 10 890 Truth : yes Truth : no Classifier: yes 100 20 Classifier: no 20 1860

Class 1 Class 2 Micro Ave. Table

• Sec. 15.2.4

Macroaveraged precision: (0.5 + 0.9)/2 = 0.7 Microaveraged precision: 100/120 = .83 Microaveraged score is dominated by score on common classes

Outline

Classification Why incorporate uncertainty Classification with Bayes Rule Example: Email Classifier Evaluation