Natural language processing with neural networks. Hubert Brykowski - - PowerPoint PPT Presentation

▶
natural language processing with neural networks
SMART_READER_LITE
LIVE PREVIEW

Natural language processing with neural networks. Hubert Brykowski - - PowerPoint PPT Presentation

Jun 21, 2023 36 likes •698 views

Natural language processing with neural networks. Hubert Brykowski Europython 2019 Hubert Brykowski hubert@brylkowski.com linkedin.com/in/hubert-bry%C5% 82kowski/ Why NLP is hard Ambiguity I had a sandwich with Bacon. By Gage Skidmore

slide-1
SLIDE 1

Natural language processing with neural networks.

Hubert Bryłkowski Europython 2019

slide-2
SLIDE 2

Hubert Bryłkowski

hubert@brylkowski.com linkedin.com/in/hubert-bry%C5% 82kowski/

slide-3
SLIDE 3

Why NLP is hard

slide-4
SLIDE 4

Ambiguity

I had a sandwich with Bacon.

By Gage Skidmore - https://www.flickr.com/photos/gageskidmore/14823923553/, CC BY-SA 2.0, https://commons.wikimedia.org/w/index.php?curid=34419969

slide-5
SLIDE 5

Ambiguity

I had a sandwich with Bacon.

slide-6
SLIDE 6

Texts are compositional

Characters -> words -> sentences -> paragraphs

slide-7
SLIDE 7

https://www.youtube.com/watch?v=LvlUBxi_JEg

slide-8
SLIDE 8

Common problems in NLP

Document classification (sentiment, author, spam)

slide-9
SLIDE 9

Common problems in NLP

Sequence to sequence (translation, summarization, response generation)

slide-10
SLIDE 10

Common problems in NLP

Information extraction (named-entity recognition) Jimmy bought an apple. Jimmy bought Apple shares. fruit company

slide-11
SLIDE 11

Why neural networks are good for NLP?

slide-12
SLIDE 12

“Real” life problem

slide-13
SLIDE 13

IMDB sentiment analysis.

25,000 highly polar movie reviews

Andrew L. Maas, Raymond E. Daly, Peter T. Pham, Dan Huang, Andrew Y. Ng, and Christopher Potts. (2011). Learning Word Vectors for Sentiment Analysis. The 49th Annual Meeting of the Association for Computational Linguistics (ACL 2011).

slide-14
SLIDE 14

Task definition

Movie review

Neural Network

slide-15
SLIDE 15

Task definition

Movie review

Neural Network

slide-16
SLIDE 16

Text as input

“A big disappointment for what was touted as an incredible film. Incredibly bad. Very pretentious. It would be nice if just once someone would create a high profile role for a young woman that was not (...)”

slide-17
SLIDE 17

Possible features

A quick brown fox.

slide-18
SLIDE 18

Possible features

A quick brown fox.

slide-19
SLIDE 19

Possible features

A quick brown fox. noun

slide-20
SLIDE 20

Possible features

A quick brown fox. noun canine

slide-21
SLIDE 21

Possible features

A quick brown fox. noun canine stem - fox lemma - fox

slide-22
SLIDE 22

Possible features

A quick brown fox. noun canine stem - fox lemma - fox TFIDF

slide-23
SLIDE 23

Bag of words

A quick brown fox.

vocab X fox 1 brown 1

  • ver

quick 1 a 1 jumps dog lazy <UNK>

slide-24
SLIDE 24

Fully connected neural network

By Glosser.ca - Own work, Derivative of File:Artificial neural network.svg, CC BY-SA 3.0, https://commons.wikimedia.org/w/ index.php?curid=24913461

slide-25
SLIDE 25

Simple model

slide-26
SLIDE 26
slide-27
SLIDE 27
slide-28
SLIDE 28

Pros and cons of FC with BoW

  • Simple - cheap and fast to train
  • Always looking at whole text
  • Kinda interpretable
  • Can’t get close to state of the art
  • Order of words do not matter
slide-29
SLIDE 29

Bag of words

I loved the movie, but cinema was terrible. I loved cinema, but the movie was terrible.

slide-30
SLIDE 30

Sequence of one-hot vectors

A quick brown fox.

vocab X fox 0 0 0 1 brown 0 0 1 0

  • ver

0 0 0 0 quick 0 1 0 0 a 1 0 0 0 jumps 0 0 0 0 dog 0 0 0 0 lazy 0 0 0 0 <UNK> 0 0 0 0

slide-31
SLIDE 31

Sequence of one-hot vectors

A quick brown vixen.

vocab X fox 0 0 0 0 brown 0 0 1 0

  • ver

0 0 0 0 quick 0 1 0 0 a 1 0 0 0 jumps 0 0 0 0 dog 0 0 0 0 lazy 0 0 0 0 <UNK> 0 0 0 1

slide-32
SLIDE 32

Sequence of one-hot vectors

A quick brown vixen.

vocab X fox brown 1

  • ver

quick 1 a 1 jumps dog lazy <NOUN> 1 <ADJ>

slide-33
SLIDE 33
slide-34
SLIDE 34

Sequence of one-hot vectors

A quick brown vixen.

vocab X fox brown 1

  • ver

quick 1 a 1 lazy <UNK> 1 <NOUN> 1 <ADJ> 1 1 <DET> 1

slide-35
SLIDE 35

Sequence of embeddings

A quick brown vixen.

vocab X

word

0.01 0.84 -0.54 0.03 0.18 0.96 -0.45 0.98

  • 0.63 -0.21 -0.82 -0.60

0.94 -0.37 0.72 0.69

Part of speech

0.20 -0.38 0.90 0.11 0.43 0.70 -0.91 -0.97

slide-36
SLIDE 36
slide-37
SLIDE 37

Pros and cons of FC with sequence

  • Still simple - cheap and fast to train
  • Order of words matter
  • Kinda interpretable
  • Can’t get close to state of the art (0.96 -

GraphStar)

  • Words at given position matter more
  • Negations are hard to catch

Deep learning course - Andrew Ng

slide-38
SLIDE 38

This movie was not good.

slide-39
SLIDE 39

This movie was not_good.

slide-40
SLIDE 40

Convolutional Neural Networks - CNNs

slide-41
SLIDE 41
slide-42
SLIDE 42
slide-43
SLIDE 43
slide-44
SLIDE 44
slide-45
SLIDE 45
slide-46
SLIDE 46
slide-47
SLIDE 47

Pros and cons of CNNs

  • Parallelize nicely - inference can be fast
  • Order of words matter
  • Positions of words matter
  • We can look at whole sentence
  • Connections can only be made between

close neighbours Understanding Convolutional Neural Networks for NLP - DENNY BRITZ

slide-48
SLIDE 48

Recurrent Neural Networks - RNNs

slide-49
SLIDE 49

This movie was not good.

slide-50
SLIDE 50

This movie was not good.

slide-51
SLIDE 51

This movie was not good.

slide-52
SLIDE 52

This movie was not good.

slide-53
SLIDE 53

This movie was not good.

slide-54
SLIDE 54

This movie was not good.

FC PREDICTION

slide-55
SLIDE 55

This movie was not good.

FC PREDICTION

slide-56
SLIDE 56

Terrible, I loved her previous movies.

slide-57
SLIDE 57

Terrible, I loved her previous movies.

slide-58
SLIDE 58

Terrible, I loved her previous movies.

FC PREDICTION

slide-59
SLIDE 59

Pros and cons of simple RNNs

  • Can give better results
  • We look at whole sequence
  • Hard to train - a lot of resources and time

needed

  • Prone to “forgetting” words from

beginning (or end) of sequence Stanford lecture Recurrent Neural Networks and Language Models

slide-60
SLIDE 60

LSTM / GRU

slide-61
SLIDE 61

By Guillaume Chevalier - Own work, CC BY 4.0, https://commons.wikimedia.org/w/index.php?curid=71836793

slide-62
SLIDE 62

By Jeblad - Own work, CC BY-SA 4.0, https://commons.wikimedia.org/w/index.php?curid=66225938

slide-63
SLIDE 63

Pros and cons of LSTM / GRU

  • Can give best results
  • Always look at whole sequence
  • Can “remember” the words from beginning
  • Hardest to train - a lot of resources and

time needed

  • Not counting transformer - best models

Stanford lecture Machine Translation and Advanced Recurrent LSTMs and GRUs Understanding LSTM Networks

slide-64
SLIDE 64

Summary

architecture accuracy 1 epoch time fully Connected with bow 0.89 2s fully connected - embeddings 0.89 1s fully connected - pos instead unk 0.88 5s fully connected - pos embeddings 0.88 3s simple RNN - embeddings 0.85 42s simple biRNN - embeddings 0.87 137s LSTM 0.88 137s https://colab.research.google.com/drive/1J3VyPNiLQ-SpA_HBw29HRjv8Oa1Ls3zJ

slide-65
SLIDE 65
slide-66
SLIDE 66

Thank you

hubert@brylkowski.com linkedin.com/in/hubert-bry%C5%82kowski/