Lecture 8: NLP and Word Embeddings Alireza Akhavan Pour - - PowerPoint PPT Presentation

Lecture 8 -

SRTTU – A.Akhavan

1

،هبنش۱۹ نابآ۱۳۹۷

Lecture 8: NLP and Word Embeddings

Alireza Akhavan Pour CLASS.VISION

Lecture 8 -

SRTTU – A.Akhavan

NLP and Word Embeddings Word representation

2

،هبنش۱۹ نابآ۱۳۹۷

V = [a, aaron, ..., zulu, <UNK>]

1-hot representation

𝑾 = 𝟐𝟏, 𝟏𝟏𝟏

I want a glass of orange ______ . I want a glass of apple ______ . 𝑃5391 𝑃9853 𝑘𝑣𝑗𝑑𝑓 ?

؟لکشم

تسا ناسکی اهرادرب مامت یسدیلقا هلصاف. تسا هدید شزومآ رد هک یتاملک یور زا دناوتیمن generalizeدنک.

Lecture 8 -

SRTTU – A.Akhavan

Featurized representation: word embedding

3

،هبنش۱۹ نابآ۱۳۹۷

تیسنج-11-0.950.970.000.01 یتنطلس0.940.93-0.010.000.020.01 نس0.710.690.03-0.020.020.03 یکاروخ0.020.000.96-0.97-0.010.01 زیاس هدنز تمیق لعف ....

300

...

𝒇𝟕𝟑𝟔𝟖

...

𝒇𝟓𝟔𝟕 I want a glass of orange ______ . I want a glass of apple ______ . 𝑘𝑣𝑗𝑑𝑓 𝑘𝑣𝑗𝑑𝑓

Lecture 8 -

SRTTU – A.Akhavan

Visualizing word embeddings

4

،هبنش۱۹ نابآ۱۳۹۷

[van der Maaten and Hinton., 2008.Visualizing data using t-SNE]

Lecture 8 -

SRTTU – A.Akhavan

Using word embeddings: Named entity recognition example

5

،هبنش۱۹ نابآ۱۳۹۷

Robert Lin is an apple farmer Sally Johnson is an

• range

farmer Robert Lin is a durian cultivator

Lecture 8 -

SRTTU – A.Akhavan

Now if you have tested your model with this sentence "Robert Lin is a durian cultivator“ the network should learn the name even if it hasn't seen the word durian before (during training). That's the power of word representations. The algorithms that are used to learn word embeddings can examine billions of words of unlabeled text - for example, 100 billion words and learn the representation from them.

6

،هبنش۱۹ نابآ۱۳۹۷

Using word embeddings: Named entity recognition example

Lecture 8 -

SRTTU – A.Akhavan

Transfer learning and word embeddings

I. Learn word embeddings from large text corpus (1-100 billion of words).

• Or download pre-trained embedding online.

II. Transfer embedding to new task with the smaller training set (say, 100k words). III. Optional: continue to finetune the word embeddings with new data.

• You bother doing this if your smaller training set (from step 2) is big enough.

7

،هبنش۱۹ نابآ۱۳۹۷

تسا یدورو داعبا شهاک رگید تبثم یگژیو کی . روتکو یاج هب لبثم10.000 یدعب

• ne-hot

روتکو کی اب300درک میهاوخ راک یدعب.

Lecture 8 -

SRTTU – A.Akhavan

Relation to face encoding (Embeddings)

8

،هبنش۱۹ نابآ۱۳۹۷

[Taigman et. al., 2014. DeepFace: Closing the gap to human level performance]

 Word embeddings have an interesting relationship to the face recognition task:

• In this problem, we encode each face into a vector and then check how similar

are these vectors.

• Words encoding and embeddings have a similar meaning here.

 In the word embeddings task, we are learning a representation for each word in our vocabulary (unlike in image encoding where we have to map each new image to some n-dimensional vector).

Lecture 8 -

SRTTU – A.Akhavan

Properties of word embeddings

• Analogies

9

،هبنش۱۹ نابآ۱۳۹۷

[Mikolov et. al., 2013, Linguistic regularities in continuous space word representations]  Can we conclude this relation:

• Man ==> Woman
• King ==> ??

𝒇𝑵𝒃𝒐 𝒇𝑿𝒑𝒏𝒃𝒐 𝒇𝑳𝒋𝒐𝒉 𝒇𝑹𝒗𝒇𝒇𝒐 eMan – eWoman ≈ −2 eKing – eQ𝐯𝐟𝐟𝐨 ≈ −2

Lecture 8 -

SRTTU – A.Akhavan

Analogies using word vectors

10

،هبنش۱۹ نابآ۱۳۹۷

300 D

man woman King Queen 𝑮𝒋𝒐𝒆 𝒙𝒑𝒔𝒆 𝒙: 𝑏𝑠𝑕 max

𝑥

൯ 𝑡𝑗𝑛(𝑓𝑥, 𝑓𝑙𝑗𝑜𝑕 − 𝑓𝑛𝑏𝑜 + 𝑓𝑥𝑝𝑛𝑏𝑜 𝒇𝒙 t-SNE

Lecture 8 -

SRTTU – A.Akhavan

Cosine similarity

11

،هبنش۱۹ نابآ۱۳۹۷

𝑡𝑗𝑛 𝑣, 𝑤 = 𝑉𝑈𝑊 𝑣 2 𝑤 2 ൯ 𝑡𝑗𝑛(𝑓𝑥, 𝑓𝑙𝑗𝑜𝑕 − 𝑓𝑛𝑏𝑜 + 𝑓𝑥𝑝𝑛𝑏𝑜 𝑊 𝑉 𝑊 𝑉 1

• 1

𝑉 − 𝑊 2 یسدیلقا هلصاف: Ottawa:Canada as Iran:Tehran Man:Woman as boy:girl Big:bigger as tall:taller Yen: Japan as Ruble:Russia

Lecture 8 -

SRTTU – A.Akhavan

Embedding matrix

12

،هبنش۱۹ نابآ۱۳۹۷

here …

• range

example ... <UNK>

• 0.2

…

• 0.67
• 0.2

… 0.2 0.7 … 0.3

• 0.5

… 0.1 0.85 … 0.25 0.3 … 1

• 0.04

…

• 0.18

0.33 …

• 0.1

... 0.5 … 1 0.3 … 0.2

300 10.000 6257 . : 1 . : 6257 10.000 𝑃6257 𝐹

Lecture 8 -

SRTTU – A.Akhavan

Embedding matrix

13

،هبنش۱۹ نابآ۱۳۹۷

here …

• range

example ... <UNK>

• 0.2

…

• 0.67
• 0.2

… 0.2 0.7 … 0.3

• 0.5

… 0.1 0.85 … 0.25 0.3 … 1

• 0.04

…

• 0.18

0.33 …

• 0.1

... 0.5 … 1 0.3 … 0.2

300 10.000 6257 . : 1 . : 6257 10.000 𝑃6257 𝐹 𝑭 . 𝑷𝟕𝟑𝟔𝟖 = 𝒇𝟕𝟑𝟔𝟖

Lecture 8 -

SRTTU – A.Akhavan

Embedding matrix

14

،هبنش۱۹ نابآ۱۳۹۷

here …

• range

example ... <UNK>

• 0.2

…

• 0.67
• 0.2

… 0.2 0.7 … 0.3

• 0.5

… 0.1 0.85 … 0.25 0.3 … 1

• 0.04

…

• 0.18

0.33 …

• 0.1

... 0.5 … 1 0.3 … 0.2

𝑭 . 𝑷𝟕𝟑𝟔𝟖 = 𝒇𝟕𝟑𝟔𝟖

300x10k . : 1 . : 10k x 1 300 x 1  If O6257 is the one hot encoding of the word orange of shape (10000, 1), then np.dot(E,O6257) = e6257 which shape is (300, 1).  Generally np.dot(E, Oj) = ej

(embedding for word j)

Lecture 8 -

SRTTU – A.Akhavan

Embedding matrix

15

،هبنش۱۹ نابآ۱۳۹۷

The Embedding layer is best understood as a dictionary mapping integer indices (which stand for specific words) to dense vectors. It takes as input integers, it looks up these integers into an internal dictionary, and it returns the associated vectors. It's effectively a dictionary lookup. https://keras.io/layers/embeddings/

Lecture 8 -

SRTTU – A.Akhavan

16

،هبنش۱۹ نابآ۱۳۹۷

عبانم

• https://www.coursera.org/specializations/deep
• learning
• https://github.com/fchollet/deep-learning-

with-python-notebooks/blob/master/6.1-using- word-embeddings.ipynb

• https://github.com/mbadry1/DeepLearning.ai-

Summary/