Social Media & Text Analysis lecture 5 - POS/NE Tagging CSE - - PowerPoint PPT Presentation

▶
social media text analysis
SMART_READER_LITE
LIVE PREVIEW

Social Media & Text Analysis lecture 5 - POS/NE Tagging CSE - - PowerPoint PPT Presentation

Nov 02, 2022 557 likes •1.17k views

Social Media & Text Analysis lecture 5 - POS/NE Tagging CSE 5539-0010 Ohio State University Instructor: Alan Ritter Website: socialmedia-class.org NLP Pipeline (summary so far) classification Regular (Nave Bayes) Expression Part-of-

slide-1
SLIDE 1

Social Media & Text Analysis

lecture 5 - POS/NE Tagging

CSE 5539-0010 Ohio State University Instructor: Alan Ritter Website: socialmedia-class.org

slide-2
SLIDE 2

Alan Ritter ◦ socialmedia-class.org

Language Identification Tokenization Part-of- Speech (POS) Tagging Shallow Parsing (Chunking) Named Entity Recognition (NER)

NLP Pipeline (summary so far)

Stemming

Normalization

classification (Naïve Bayes) Regular Expression

slide-3
SLIDE 3

Alan Ritter ◦ socialmedia-class.org

Language Identification Tokenization Part-of- Speech (POS) Tagging Shallow Parsing (Chunking) Named Entity Recognition (NER)

NLP Pipeline (next)

Sequential Tagging

Stemming

Normalization

slide-4
SLIDE 4

Challenge: Natural Language Processing Breaks

4

slide-5
SLIDE 5

Challenge: Natural Language Processing Breaks

4

LOCATION PERSON

slide-6
SLIDE 6

Challenge: Natural Language Processing Breaks

4

LOCATION PERSON

0.00 0.10 0.20 0.30 0.40 0.50 0.60 0.70 0.80 0.90 1.00

Newswire Twitter

Stanford NER: ~50% Drop

slide-7
SLIDE 7

Alan Ritter ◦ socialmedia-class.org

Part-of-Speech (POS) Tagging

Cant MD wait VB for IN the DT ravens NNP game NN tomorrow NN … : go VB ray NNP rice NNP !!!!!!! .

slide-8
SLIDE 8

Alan Ritter ◦ socialmedia-class.org

Penn Treebank POS Tags

slide-9
SLIDE 9

Alan Ritter ◦ socialmedia-class.org

Part-of-Speech (POS) Tagging

  • Words often have more than one POS:
  • The back door = JJ
  • On my back = NN
  • Win the voters back = RB
  • Promised to back the bill = VB
  • POS tagging problem is to determine the POS tag

for a particular instance of a word.

Source: adapted from Chris Manning

slide-10
SLIDE 10

Alan Ritter ◦ socialmedia-class.org

Twitter-specific Tags

  • #hashtag
  • @metion
  • url
  • email address
  • emoticon
  • discourse marker
  • symbols
  • …

Source: Gimpel et al. 
 “Part-of-Speech Tagging for Twitter : Annotation, Features, and Experiments” ACL 2011

slide-11
SLIDE 11

Alan Ritter ◦ socialmedia-class.org

Noisy Text: Challenges

  • Lexical Variation (misspellings, abbreviations)

– `2m', `2ma', `2mar', `2mara', `2maro', `2marrow', `2mor', `2mora', `2moro', `2morow', `2morr', `2morro', `2morrow', `2moz', `2mr', `2mro', `2mrrw', `2mrw', `2mw', `tmmrw', `tmo', `tmoro', `tmorrow', `tmoz', `tmr', `tmro', `tmrow', `tmrrow', `tmrrw', `tmrw', `tmrww', `tmw', `tomaro', `tomarow', `tomarro', `tomarrow', `tomm', `tommarow', `tommarrow', `tommoro', `tommorow', `tommorrow', `tommorw', `tommrow', `tomo', `tomolo', `tomoro', `tomorow', `tomorro', `tomorrw', `tomoz', `tomrw', `tomz‘

  • Unreliable Capitalization

– “The Hobbit has FINALLY started filming! I cannot wait!”

  • Unique Grammar

– “watchng american dad.”

7

slide-12
SLIDE 12

Alan Ritter ◦ socialmedia-class.org

Chunking

Cant VP wait for PP the NP ravens game tomorrow NP … go VP ray NP rice !!!!!!!

slide-13
SLIDE 13

Alan Ritter ◦ socialmedia-class.org

Chunking

  • recovering phrases constructed by the part-of-speech

tags

  • a.k.a shallow (partial) parsing:
  • full parsing is expensive, and is not very robust
  • partial parsing can be much faster, more robust, yet

sufficient for many applications

  • useful as input (features) for named entity

recognition or full parser

slide-14
SLIDE 14

Alan Ritter ◦ socialmedia-class.org

Named Entity Recognition(NER)

Cant wait for the ravens ORG game tomorrow … go ray PER rice !!!!!!! .

ORG: organization PER: person LOC: location

slide-15
SLIDE 15

Alan Ritter ◦ socialmedia-class.org

Cant wait for the ravens ORG game tomorrow … go ray PER rice !!!!!!! .

ORG: organization PER: person LOC: location

NER: Basic Classes

slide-16
SLIDE 16

Noisy Text: NLP breaks

POS: Chunk: NER:

slide-17
SLIDE 17

Noisy Text: NLP breaks

POS: Chunk: NER:

slide-18
SLIDE 18

Noisy Text: NLP breaks

POS: Chunk: NER:

slide-19
SLIDE 19

Noisy Text: NLP breaks

POS: Chunk: NER:

slide-20
SLIDE 20

Noisy Text: NLP breaks

POS: Chunk: NER:

Noisy Style

slide-21
SLIDE 21

Alan Ritter ◦ socialmedia-class.org

NER: Rich Classes

Source: Strauss, Toma, Ritter, de Marneffe, Xu 
 Results of the WNUT16 Named Entity Recognition Shared Task (WNUT@COLING 2016)

slide-22
SLIDE 22

Alan Ritter ◦ socialmedia-class.org

NER: Genre Differences

News Tweets PER Politicians, business leaders, journalists, celebrities Sportsmen, actors, TV personalities, celebrities, names of friends LOC Countries, cities, rivers, and other places related to current affairs Restaurants, bars, local landmarks/areas, cities, rarely countries ORG Public and private companies, government

  • rganisations

Bands, internet companies, sports clubs

Source: Kalina Bontcheva and Leon Derczynski 
 “Tutorial on Natural Language Processing for Social Media” EACL 2014

slide-23
SLIDE 23
  • Freebase / Wikipedia lists provide a source of

supervision

  • But these lists are highly ambiguous
  • Example: China

Weakly Supervised NER

slide-24
SLIDE 24
  • Freebase / Wikipedia lists provide a source of

supervision

  • But these lists are highly ambiguous
  • Example: China

Weakly Supervised NER

slide-25
SLIDE 25
  • Freebase / Wikipedia lists provide a source of

supervision

  • But these lists are highly ambiguous
  • Example: China

Weakly Supervised NER

slide-26
SLIDE 26
  • Freebase / Wikipedia lists provide a source of

supervision

  • But these lists are highly ambiguous
  • Example: China

Weakly Supervised NER

slide-27
SLIDE 27
  • Freebase / Wikipedia lists provide a source of

supervision

  • But these lists are highly ambiguous
  • Example: China

Weakly Supervised NER

…

slide-28
SLIDE 28

Distant Supervision with Latent Variables

[Ritter, et. al. EMNLP 2011]

Latent variable model for Named Entity Categorization with constraints

slide-29
SLIDE 29

Obama Apple

On my way to JFK early in the… JFK 's bomber jacket sells for… JFK Airport’s Pan Am Worldport… Waiting at JFK for our ride… When JFK threw first pitch on…

JFK

[Ritter, et. al. EMNLP 2011]

…" …"

slide-30
SLIDE 30

Obama Apple

On my way to JFK early in the… JFK 's bomber jacket sells for… JFK Airport’s Pan Am Worldport… Waiting at JFK for our ride… When JFK threw first pitch on…

JFK

‘s 0.04 threw 0.02 jacket 0.01 …

waiting 0.04 ride 0.03 way 0.02 …

announced 0.04 new 0.03 release 0.02 …

PERSON FACILITY PRODUCT

[Ritter, et. al. EMNLP 2011]

…" …"

slide-31
SLIDE 31

Obama Apple

On my way to JFK early in the… JFK 's bomber jacket sells for… JFK Airport’s Pan Am Worldport… Waiting at JFK for our ride… When JFK threw first pitch on…

JFK

1.25 2.5 3.75 5

‘s 0.04 threw 0.02 jacket 0.01 …

waiting 0.04 ride 0.03 way 0.02 …

announced 0.04 new 0.03 release 0.02 …

PERSON FACILITY PRODUCT

[Ritter, et. al. EMNLP 2011]

…" …"

slide-32
SLIDE 32

Obama Apple

On my way to JFK early in the… JFK 's bomber jacket sells for… JFK Airport’s Pan Am Worldport… Waiting at JFK for our ride… When JFK threw first pitch on…

JFK

1.25 2.5 3.75 5

‘s 0.04 threw 0.02 jacket 0.01 …

waiting 0.04 ride 0.03 way 0.02 …

announced 0.04 new 0.03 release 0.02 …

PERSON FACILITY PRODUCT

[Ritter, et. al. EMNLP 2011]

X

…" …"

slide-33
SLIDE 33

Obama Apple

On my way to JFK early in the… JFK 's bomber jacket sells for… JFK Airport’s Pan Am Worldport… Waiting at JFK for our ride… When JFK threw first pitch on…

JFK

1.25 2.5 3.75 5

‘s 0.04 threw 0.02 jacket 0.01 …

waiting 0.04 ride 0.03 way 0.02 …

announced 0.04 new 0.03 release 0.02 …

PERSON FACILITY PRODUCT X

…" …"

[Ritter, et. al. EMNLP 2011]

slide-34
SLIDE 34

Example Type Lists

[Ritter, et. al. EMNLP 2011]

slide-35
SLIDE 35

Example Type Lists

KKTNY = Kourtney and Kim Take New York RHOBH = Real Housewives of Beverly Hills

[Ritter, et. al. EMNLP 2011]

slide-36
SLIDE 36

Example Type Lists

KKTNY = Kourtney and Kim Take New York RHOBH = Real Housewives of Beverly Hills

[Ritter, et. al. EMNLP 2011]

slide-37
SLIDE 37

Twitter NER: Classification Results

0.175 0.35 0.525 0.7 Majority Baseline Freebase Baseline Supervised Baseline DL-Cotrain LLDA

(Collins and Singer ‘99)

[Ritter, et. al. EMNLP 2011]

F1

slide-38
SLIDE 38

Twitter NER: Classification Results

0.175 0.35 0.525 0.7 Majority Baseline Freebase Baseline Supervised Baseline DL-Cotrain LLDA

(Collins and Singer ‘99)

25% increase in F1

[Ritter, et. al. EMNLP 2011]

F1

slide-39
SLIDE 39

Alan Ritter ◦ socialmedia-class.org

Tool: twitter_nlp

slide-40
SLIDE 40

Alan Ritter ◦ socialmedia-class.org

Tool: twitter_nlp

slide-41
SLIDE 41

Results of the WNUT16 Named Entity Recognition Shared Task

Benjamin Strauss, Bethany Toma, Alan Ritter, Marie- Catherine de Marneffe and Wei Xu

slide-42
SLIDE 42

Need for Shared Evaluations

  • Fast Moving Area: Papers published in the same

year use different datasets and evaluation methodology

  • Performance still behind what we would like
  • ~0.6 - 0.7 F1 score (much lower than news)
  • Explore new ideas & approaches
slide-43
SLIDE 43

Related NER Evaluations

  • MUC
  • http://www.itl.nist.gov/iaui/894.02/related_projects/muc/muc_data/muc_data_index.html
  • CONLL
  • http://www.cnts.ua.ac.be/conll2002/ner/
  • http://www.cnts.ua.ac.be/conll2003/ner/
  • ACE
  • https://catalog.ldc.upenn.edu/LDC2005T09
  • Named Entity rEcognition and Linking (NEEL) Challenge
  • #Microposts workshop at WWW
  • http://microposts2016.seas.upenn.edu/challenge.html

Newswire Newswire Newswire Microblogs

slide-44
SLIDE 44

Twitter NER Evaluation Summary

Re-Run of 2015 Task 2 Subtasks

  • Segmentation + 10 way classification
  • Segmentation only (no classification)
slide-45
SLIDE 45

Twitter NER Evaluation Summary

Re-Run of 2015 Task 2 Subtasks

  • Segmentation + 10 way classification
  • Segmentation only (no classification)

New test set annotated for 2016

slide-46
SLIDE 46

Twitter NER Evaluation Summary

Re-Run of 2015 Task 2 Subtasks

  • Segmentation + 10 way classification
  • Segmentation only (no classification)

New test set annotated for 2016 10 Participating Teams

slide-47
SLIDE 47

Data

Training + Dev Data:

  • All training, dev, test data from 2015
  • Training: 2,394 tweets, Dev: 1,420 tweets
slide-48
SLIDE 48

Data

Training + Dev Data:

  • All training, dev, test data from 2015
  • Training: 2,394 tweets, Dev: 1,420 tweets

Test Data

  • 3,856 tweets
  • Later Time Period
  • No overlap in time period with Training/Dev data
slide-49
SLIDE 49

2016 Test Data Annotation

  • Simple Annotation Guidelines:
  • http://bit.ly/1FSP6i2
  • Re-annotated 100 tweets from 2015 data
  • Good agreement - 0.68 F-Score
slide-50
SLIDE 50

2016 Test Data Annotation

  • Simple Annotation Guidelines:
  • http://bit.ly/1FSP6i2
  • Re-annotated 100 tweets from 2015 data
  • Good agreement - 0.68 F-Score
  • Frequent questions, discussion

among the group

  • BRAT annotation tool
  • (http://brat.nlplab.org/)
slide-51
SLIDE 51

Annotation Interface

slide-52
SLIDE 52

10 Participating Teams

slide-53
SLIDE 53

Approaches

  • All teams use some form of machine supervised ML
  • Many LSTM-based approaches as compared to last year
  • Unique approaches: CambridgeLTL, Talos
slide-54
SLIDE 54

Results (10 types)

slide-55
SLIDE 55

Results (No Types)

slide-56
SLIDE 56

Domain-Specific Data

Cybersecurity (350 Tweets) Gun Violence (500 Tweets)

slide-57
SLIDE 57

Results (Cyber-Domain)

slide-58
SLIDE 58

Results (Shooting-Domain)

slide-59
SLIDE 59

Alan Ritter ◦ socialmedia-class.org

Language Identification Tokenization Part-of- Speech (POS) Tagging Shallow Parsing (Chunking) Named Entity Recognition (NER)

Summary

Stemming

Normalization

classification (Naïve Bayes) Regular Expression Sequential Tagging

slide-60
SLIDE 60

Alan Ritter ◦ socialmedia-class.org

Presentation 2