SLIDE 1
Naïve Bayes
CMSC 473/673 UMBC
Some slides adapted from 3SLP
Outline
Terminology: bag-of-words “Naïve” assumption Training & performance NB as a language model
Nave Bayes CMSC 473/673 UMBC Some slides adapted from 3SLP - - PowerPoint PPT Presentation
Nave Bayes CMSC 473/673 UMBC Some slides adapted from 3SLP - - PowerPoint PPT Presentation
Nave Bayes CMSC 473/673 UMBC Some slides adapted from 3SLP Outline Terminology: bag-of-words Nave assumption Training & performance NB as a language model Outline Terminology: bag-of-words Nave assumption Training
SLIDE 2
SLIDE 3
Outline
Terminology: bag-of-words “Naïve” assumption Training & performance NB as a language model
SLIDE 4
Bag-of-words
Based on some tokenization, turn an input document into an array (or dictionary or set) of its unique vocab items Position/ordering in the document is not important Bag-of-words go hand-in-hand with one-hot representations but they can be extended to handle dense representations Short-hand: BOW
SLIDE 5
The Bag of Words Representation
SLIDE 6
The Bag of Words Representation
SLIDE 7
The Bag of Words Representation
7
SLIDE 8
The Bag of Words Representation
8
Count-based BOW
SLIDE 9
Bag-of-words as a Function
Based on some tokenization, turn an input document into an array (or dictionary or set) of its unique vocab items Think of getting a BOW rep. as a function f input: Document
- utput: Container of size E, indexable by
each vocab type v
SLIDE 10
Some Bag-of-words Functions
Kind Type of 𝒈𝒘 Interpretation Binary 0, 1 Did v appear in the document? Count-based Natural number (int >= 0) How often did v occur in the document? Averaged Real number (>=0, <= 1) How often did v occur in the document, normalized by doc length? TF-IDF (term frequency, inverse document frequency) Real number (>= 0) How frequent is a word, tempered by how prevalent it is across the corpus (to be covered later!) …
SLIDE 11
Some Bag-of-words Functions
Kind Type of 𝒈𝒘 Interpretation Binary 0, 1 Did v appear in the document? Count-based Natural number (int >= 0) How often did v occur in the document? Averaged Real number (>=0, <= 1) How often did v occur in the document, normalized by doc length? TF-IDF (term frequency, inverse document frequency) Real number (>= 0) How frequent is a word, tempered by how prevalent it is across the corpus (to be covered later!) …
Q: Is this a reasonable representation? Q: What are some tradeoffs (benefits vs. costs)?
SLIDE 12
Outline
Terminology: bag-of-words “Naïve” assumption Training & performance NB as a language model
SLIDE 13
Bag of Words Classifier
γ( )=c
seen 2 sweet 1 whimsical 1 recommend 1 happy 1 ... ...
classifier classifier
SLIDE 14
Naïve Bayes (NB) Classifier argmax𝑍𝑞 𝑌 𝑍) ∗ 𝑞(𝑍)
Start with Bayes Rule
label text Q: Are we doing discriminative training
- r generative training?
SLIDE 15
Naïve Bayes (NB) Classifier
Start with Bayes Rule
label text Q: Are we doing discriminative training
- r generative training?
A: generative training
argmax𝑍𝑞 𝑌 𝑍) ∗ 𝑞(𝑍)
SLIDE 16
Naïve Bayes (NB) Classifier argmax𝑍 ෑ
𝑢
𝑞(𝑌𝑢|𝑍) ∗ 𝑞(𝑍)
Adopt naïve bag of words representation Xt
label each word Iterate through types
SLIDE 17
Naïve Bayes (NB) Classifier argmax𝑍 ෑ
𝑢
𝑞(𝑌𝑢|𝑍) ∗ 𝑞(𝑍)
Adopt naïve bag of words representation Xt Assume position doesn’t matter
label each word Iterate through types
SLIDE 18
Outline
Terminology: bag-of-words “Naïve” assumption Training & performance NB as a language model
SLIDE 19
Learning for a Naïve Bayes Classifier
Assuming V vocab types 𝑥1, … , 𝑥𝑊 and L classes 𝑣1, … , 𝑣𝑀(and appropriate corpora)
SLIDE 20
Learning for a Naïve Bayes Classifier
Assuming V vocab types 𝑥1, … , 𝑥𝑊 and L classes 𝑣1, … , 𝑣𝑀(and appropriate corpora)
Q: What parameters (values/weights) must be learned?
SLIDE 21
Learning for a Naïve Bayes Classifier
Assuming V vocab types 𝑥1, … , 𝑥𝑊 and L classes 𝑣1, … , 𝑣𝑀(and appropriate corpora)
Q: What parameters (values/weights) must be learned? A: 𝑞 𝑥𝑤 𝑣𝑚 , 𝑞(𝑣𝑚)
SLIDE 22
Learning for a Naïve Bayes Classifier
Assuming V vocab types 𝑥1, … , 𝑥𝑊 and L classes 𝑣1, … , 𝑣𝑀(and appropriate corpora)
Q: What parameters (values/weights) must be learned? A: 𝑞 𝑥𝑤 𝑣𝑚 , 𝑞(𝑣𝑚) Q: How many parameters must be learned?
SLIDE 23
Learning for a Naïve Bayes Classifier
Assuming V vocab types 𝑥1, … , 𝑥𝑊 and L classes 𝑣1, … , 𝑣𝑀(and appropriate corpora)
Q: What parameters (values/weights) must be learned? A: 𝑞 𝑥𝑤 𝑣𝑚 , 𝑞(𝑣𝑚) Q: How many parameters must be learned? A: 𝑀𝐿 + 𝑀
SLIDE 24
Learning for a Naïve Bayes Classifier
Assuming V vocab types 𝑥1, … , 𝑥𝑊 and L classes 𝑣1, … , 𝑣𝑀(and appropriate corpora)
Q: What parameters (values/weights) must be learned? A: 𝑞 𝑥𝑤 𝑣𝑚 , 𝑞(𝑣𝑚) Q: How many parameters must be learned? A: 𝑀𝐿 + 𝑀 Q: What distributions need to sum to 1?
SLIDE 25
Learning for a Naïve Bayes Classifier
Assuming V vocab types 𝑥1, … , 𝑥𝑊 and L classes 𝑣1, … , 𝑣𝑀(and appropriate corpora)
Q: What parameters (values/weights) must be learned? A: 𝑞 𝑥𝑤 𝑣𝑚 , 𝑞(𝑣𝑚) Q: How many parameters must be learned? A: 𝑀𝐿 + 𝑀 Q: What distributions need to sum to 1? A: Each 𝑞 ⋅ 𝑣𝑚 , and the prior
SLIDE 26
Learning for a Naïve Bayes Classifier
Assuming V vocab types 𝑥1, … , 𝑥𝑊 and L classes 𝑣1, … , 𝑣𝑀(and appropriate corpora) If you’re going to compute perplexity from 𝑞 ⋅ 𝑣𝑚 , all class specific language models 𝑞 ⋅ 𝑣𝑚 MUST share a common vocab (otherwise it’s not a fair comparison!!!)
SLIDE 27
Learning for a Naïve Bayes Classifier
Assuming V vocab types 𝑥1, … , 𝑥𝑊 and L classes 𝑣1, … , 𝑣𝑀(and appropriate corpora) If you’re going to compute perplexity from 𝑞 ⋅ 𝑣𝑚 , all class specific language models 𝑞 ⋅ 𝑣𝑚 MUST share a common vocab (otherwise it’s not a fair comparison!!!)
Q: Should OOV and UNK be included?
SLIDE 28
Learning for a Naïve Bayes Classifier
Assuming V vocab types 𝑥1, … , 𝑥𝑊 and L classes 𝑣1, … , 𝑣𝑀(and appropriate corpora) If you’re going to compute perplexity from 𝑞 ⋅ 𝑣𝑚 , all class specific language models 𝑞 ⋅ 𝑣𝑚 MUST share a common vocab (otherwise it’s not a fair comparison!!!)
Q: Should OOV and UNK be included? Q: Should EOS be included?
SLIDE 29
Learning for a Naïve Bayes Classifier
Assuming V vocab types 𝑥1, … , 𝑥𝑊 and L classes 𝑣1, … , 𝑣𝑀(and appropriate corpora) If you’re going to compute perplexity from 𝑞 ⋅ 𝑣𝑚 , all class specific language models 𝑞 ⋅ 𝑣𝑚 MUST share a common vocab (otherwise it’s not a fair comparison!!!) A binary/count-based NB classifier is also called a Multinomial Naïve Bayes classifier
SLIDE 30
Multinomial Naïve Bayes: Learning
From training corpus, extract Vocabulary
SLIDE 31
Multinomial Naïve Bayes: Learning
Calculate P(cj) terms For each cjin C do
docsj = all docs with class =cj
From training corpus, extract Vocabulary
𝑞 𝑑
𝑘 = |𝑒𝑝𝑑𝑡 𝑘|
# 𝑒𝑝𝑑𝑡
SLIDE 32
Multinomial Naïve Bayes: Learning
Calculate P(cj) terms For each cjin C do
docsj = all docs with class =cj
Calculate P(wk | cj) terms Textj = single doc containing all docsj Foreach word wk in Vocabulary nk = # of occurrences of wk in Textj
From training corpus, extract Vocabulary
𝑞 𝑑
𝑘 = |𝑒𝑝𝑑𝑡 𝑘|
# 𝑒𝑝𝑑𝑡 𝑞 𝑥𝑙| 𝑑𝑘 = class unigram LM
SLIDE 33
Brill and Banko (2001) With enough data, the classifier may not matter
SLIDE 34
Outline
Terminology: bag-of-words “Naïve” assumption Training & performance NB as a language model
SLIDE 35
Naïve Bayes as a Language Model
Sec.13.2.1
0.1 I 0.1 love 0.01 this 0.05 fun 0.1 film
Positive Model Negative Model
0.2 I 0.001 love 0.01 this 0.005 fun 0.1 film
SLIDE 36
Naïve Bayes as a Language Model
Which class assigns the higher probability to s?
film love this fun I
Sec.13.2.1
0.1 I 0.1 love 0.01 this 0.05 fun 0.1 film
Positive Model Negative Model
0.2 I 0.001 love 0.01 this 0.005 fun 0.1 film
SLIDE 37
Naïve Bayes as a Language Model
Which class assigns the higher probability to s?
0.1 I 0.1 love 0.01 this 0.05 fun 0.1 film
Positive Model Negative Model
film love this fun I
0.1 0.1 0.01 0.05 0.1 0.1 0.001 0.01 0.005 0.2
0.2 I 0.001 love 0.01 this 0.005 fun 0.1 film
Sec.13.2.1
SLIDE 38
Naïve Bayes as a Language Model
Which class assigns the higher probability to s?
0.1 I 0.1 love 0.01 this 0.05 fun 0.1 film
Positive Model Negative Model
film love this fun I
0.1 0.1 0.01 0.05 0.1 0.1 0.001 0.01 0.005 0.2
5e-7 ≈ P(s|pos) > P(s|neg) ≈ 1e-9
0.2 I 0.001 love 0.01 this 0.005 fun 0.1 film
Sec.13.2.1
SLIDE 39
Naïve Bayes To Try
http://csee.umbc.edu/courses/undergraduate/473/f19/nb
- Toy problem: classify whether a tweet will be retweeted
- Toy problem: OOV and EOS are not included
- Laplace smoothing is used for p(word | label)
SLIDE 40
Summary: Naïve Bayes is Not So Naïve
Very Fast, low storage requirements Robust to Irrelevant Features Very good in domains with many equally important features Optimal if the independence assumptions hold Dependable baseline for text classification (but often not the best)
SLIDE 41
But: Naïve Bayes Isn’t Without Issue
Model the posterior in one go? Are the features really uncorrelated? Are plain counts always appropriate? Are there “better” ways of handling missing/noisy data? (automated, more principled)
SLIDE 42
Outline
Terminology: bag-of-words “Naïve” assumption Training & performance NB as a language model