word embeddings rappel embeddings pas word embeddings est
play

Word embeddings Rappel Embeddings ( pas Word Embeddings ) Est une - PowerPoint PPT Presentation

Feb 28, 2024 •296 likes •780 views

Word embeddings Rappel Embeddings ( pas Word Embeddings ) Est une lookup table Formalisme: Index dun mot: w i Table dembeddings (lookup matrix): V Embedding: e i e i = V( w i ) Reprsentation dun mot

  1. Word embeddings

  2. Rappel Embeddings ( pas Word Embeddings )

  3. Est une “lookup table” Formalisme: ● Index d’un mot: w i ● Table d’embeddings (lookup matrix): V ● Embedding: e i ● e i = V( w i )

  4. Représentation d’un mot Différentes possibilités: Vecteur One-hot ● ○ Chat : [0,0,… 0, 1 ,0,0,0,0,0,0,0,0,0…] ● Vecteur de context ○ Chat : [ 1 ,0,… 0, 0 ,0,0,0, 1 ,0,0, 1 ,0,0…] félin litière lait chat

  5. Vecteurs de contexte “Chap. 15: Vector Semantics.” Speech and Language Processing: an Introduction to Natural Language Processing, Computational L inguistics, and Speech Recognition, by Dan Jurafsky and James H. Martin, Dorling Kindersley Pvt, Ltd., 2014.

  6. Vecteurs de contexte “Chap. 15: Vector Semantics.” Speech and Language Processing: an Introduction to Natural Language Processing, Computational L inguistics, and Speech Recognition, by Dan Jurafsky and James H. Martin, Dorling Kindersley Pvt, Ltd., 2014.

  7. Vecteurs de contexte “Chap. 15: Vector Semantics.” Speech and Language Processing: an Introduction to Natural Language Processing, Computational L inguistics, and Speech Recognition, by Dan Jurafsky and James H. Martin, Dorling Kindersley Pvt, Ltd., 2014.

  8. Vecteurs de contexte “Chap. 15: Vector Semantics.” Speech and Language Processing: an Introduction to Natural Language Processing, Computational L inguistics, and Speech Recognition, by Dan Jurafsky and James H. Martin, Dorling Kindersley Pvt, Ltd., 2014.

  9. Vecteurs de contexte Vecteurs très grands (taille du vocabulaire) ● Contiennent beaucoup de 0 ● ● On cherche donc une manière de réduire la dimensionnalité pour : ○ Efficacité en mémoire ○ Facile d’utilisation pour des classificateurs Moins de paramètres ○ Des dimensions peuvent se recouper ○

  10. Décomposition en valeurs singulières “Chap. 15: Vector Semantics.” Speech and Language Processing: an Introduction to Natural Language Processing, Computational L inguistics, and Speech Recognition, by Dan Jurafsky and James H. Martin, Dorling Kindersley Pvt, Ltd., 2014.

  11. On conserve les top k valeurs singulières “Chap. 15: Vector Semantics.” Speech and Language Processing: an Introduction to Natural Language Processing, Computational L inguistics, and Speech Recognition, by Dan Jurafsky and James H. Martin, Dorling Kindersley Pvt, Ltd., 2014.

  12. On utilise ensuite seulement la matrice W “Chap. 15: Vector Semantics.” Speech and Language Processing: an Introduction to Natural Language Processing, Computational L inguistics, and Speech Recognition, by Dan Jurafsky and James H. Martin, Dorling Kindersley Pvt, Ltd., 2014.

  13. Méthodes à réseaux de neurones

  14. GloVe “A weighted least squares regression model” L’idée est de prédire le nombre de co -occurrences X ij (ou le log ) des mots w i et w j S’apparente à Word2Vec (ou encore FastText)

  15. GloVe w j w i 25

  16. GloVe Perte = v( w i ) * v( w j ) + b i + b j - log (X ij )

  17. GloVe Perte = v( w i ) * v( w j ) + b i + b j - log (25)

  18. Word2Vec 2 algorithmes: Skip-Gram ● CBOW (Contextual Bag of Words) ●

  19. Word2Vec

  20. CBOW

  21. CBOW

  22. CBOW - Negative Sampling litière chat softmax

  23. CBOW - Negative Sampling café chat litière pomme feuille

  24. CBOW - Negative Sampling score entre un mot w et un context C

  25. CBOW - Comment obtenir un score Produit vectoriel entre v C et v w

  26. CBOW - Comment obtenir un score = <wh, whe, her, ere, re>, <where>

  27. CBOW - Phrase Representations v (New) + v (York) ≈ Boston?

  28. CBOW - Phrase Representations v (New) + v (York) ≈ Issshhh?

  29. CBOW - Phrase Representations New York => New_York

  30. Démo FastText

  31. Recap FastText … le petit chat saute sur ...

  32. Recap FastText le petit saute sur [-2.2, 2.3, 2.4] [-0.2, -1.3, 0.4] [-3.2, 1.3, 0.5] [-3.2, 1.3, 0.5] chat [0.2, 1.3, 3.4] w i c 1 c 2 c 3 c 4 C Negative sampling: score + marteau [1.2, -1.3, -3.4] - n i score

  33. ELMo On le verra dans la section modèles de langue..!

  34. Vecteurs de phrases

  35. Comment obtenir la représentation d’une phrase? Prendre la moyenne des embeddings de mots ● Utiliser une idée similaire à Skip-Gram! ●

  36. Skip-Thought Vectors Idée de base: Étant donné un triplet de phrases ( s i-1 , s i , s i+1 ) ● Encoder la phrase s i ○ ○ Générer les phrases s i-1 et s i+1

  37. Skip-Thought Vectors

  38. Skip-Thought Vectors

  39. Skip-Thought Vectors

  40. Skip-Thought Vectors

  41. Skip-Thought Vectors

  42. Skip-Thought Vectors

  43. Skip-Thought Vectors

  44. Skip-Thought Vectors

  45. Skip-Thought Vectors Modèles de langue

  46. Skip-Thought Vectors Au final, on se sert de ça!

  47. Skip-Thought Vectors Probabilité d’avoir généré la phrase suivante Probabilité d’avoir généré la phrase précédente

Download Presentation
Download Policy: The content available on the website is offered to you 'AS IS' for your personal information and use only. It cannot be commercialized, licensed, or distributed on other websites without prior consent from the author. To download a presentation, simply click this link. If you encounter any difficulties during the download process, it's possible that the publisher has removed the file from their server.

Recommend

Word Embeddings Natural Language Processing VU (706.230) - Andi Rexha 02/04/2020 Word Embeddings

Word Embeddings Natural Language Processing VU (706.230) - Andi Rexha 02/04/2020 Word Embeddings

Word Embeddings Natural Language Processing VU (706.230) - Andi Rexha 02/04/2020 Word Embeddings Agenda Traditional NLP Word Embeddings-1 Word Embeddings-2 Text preprocessing Topic Modeling ELMo Bag-of-words model Neural

900 views • 64 slides
Word Embeddings CS 6956: Deep Learning for NLP Overview Representing meaning Word

Word Embeddings CS 6956: Deep Learning for NLP Overview Representing meaning Word

Word Embeddings CS 6956: Deep Learning for NLP Overview Representing meaning Word embeddings: Early work Word embeddings via language models Word2vec and Glove Evaluating embeddings Design choices and open questions 1

422 views • 17 slides
Word Embeddings CS 6956: Deep Learning for NLP Overview Representing meaning Word

Word Embeddings CS 6956: Deep Learning for NLP Overview Representing meaning Word

Word Embeddings CS 6956: Deep Learning for NLP Overview Representing meaning Word embeddings: Early work Word embeddings via language models Word2vec and Glove Evaluating embeddings Design choices and open questions 1

1.25k views • 75 slides
Word Embeddings CS 6956: Deep Learning for NLP Overview Representing meaning Word

Word Embeddings CS 6956: Deep Learning for NLP Overview Representing meaning Word

Word Embeddings CS 6956: Deep Learning for NLP Overview Representing meaning Word embeddings: Early work Word embeddings via language models Word2vec and Glove Evaluating embeddings Design choices and open questions 1

828 views • 15 slides
Word Embeddings CS 6956: Deep Learning for NLP Overview Representing meaning Word

Word Embeddings CS 6956: Deep Learning for NLP Overview Representing meaning Word

Word Embeddings CS 6956: Deep Learning for NLP Overview Representing meaning Word embeddings: Early work Word embeddings via language models Word2vec and Glove Evaluating embeddings Design choices and open questions 1

2k views • 24 slides
Mixed membership word embeddings: Corpus-specific embeddings without big data James Foulds

Mixed membership word embeddings: Corpus-specific embeddings without big data James Foulds

Mixed membership word embeddings: Corpus-specific embeddings without big data James Foulds University of California, San Diego Southern California Machine Learning Symposium, Caltech, 11/18/2018 Word Embeddings Language models which learn

537 views • 32 slides
Word Embeddings Revisited: Contextual Embeddings CS 6956: Deep Learning for NLP Overview

Word Embeddings Revisited: Contextual Embeddings CS 6956: Deep Learning for NLP Overview

Word Embeddings Revisited: Contextual Embeddings CS 6956: Deep Learning for NLP Overview Word types and tokens Training contextual embeddings Embeddings from Language Models (ELMo) 1 Overview Word types and tokens Training

670 views • 40 slides
Word Embeddings Tutorial HILA GONEN PHD STUDENT AT YOAV GOLDBERGS LAB BAR ILAN UNIVERSITY

Word Embeddings Tutorial HILA GONEN PHD STUDENT AT YOAV GOLDBERGS LAB BAR ILAN UNIVERSITY

Word Embeddings Tutorial HILA GONEN PHD STUDENT AT YOAV GOLDBERGS LAB BAR ILAN UNIVERSITY 5/3/18 Outline NLP Intro Word representations and word embeddings Word2vec models Visualizing word embeddings Word2vec in

1.18k views • 63 slides
Word Embeddings through Hellinger PCA Rmi Lebret and Ronan Collobert Idiap Research Institute /

Word Embeddings through Hellinger PCA Rmi Lebret and Ronan Collobert Idiap Research Institute /

Word Embeddings through Hellinger PCA Rmi Lebret and Ronan Collobert Idiap Research Institute / EPFL EACL, 29 April 2014 Word Embeddings Continuous vector-space models. Represent word meanings with vectors capturing semantic +

831 views • 46 slides
Searching for the X-Factor: Exploring Corpus Subjectivity for Word Embeddings Maksim Tkachenko

Searching for the X-Factor: Exploring Corpus Subjectivity for Word Embeddings Maksim Tkachenko

Searching for the X-Factor: Exploring Corpus Subjectivity for Word Embeddings Maksim Tkachenko and Chong Cher Chia and Hady W. Lauw Singapore Management University Word Embeddings Dense vectors of words Unsupervised training: GloVe,

522 views • 19 slides
Dense Word Embeddings CMSC 470 Marine Carpuat Slides credit: Jurasky &amp; Martin How to

Dense Word Embeddings CMSC 470 Marine Carpuat Slides credit: Jurasky &amp; Martin How to

Dense Word Embeddings CMSC 470 Marine Carpuat Slides credit: Jurasky &amp; Martin How to generate vector embeddings? One approach: feedforward neural language models Training a neural language model just to get word embeddings is expensive!

569 views • 38 slides
Dense Word Embeddings CMSC 470 Marine Carpuat Slides credit: Jurasky &amp; Martin How to

Dense Word Embeddings CMSC 470 Marine Carpuat Slides credit: Jurasky &amp; Martin How to

Dense Word Embeddings CMSC 470 Marine Carpuat Slides credit: Jurasky &amp; Martin How to generate vector embeddings? One approach: feedforward neural language models Training a neural language model just to get word embeddings is expensive!

537 views • 35 slides
Pattern-based Solutions to Limitations of Leading Word Embeddings Roy Schwartz University of

Pattern-based Solutions to Limitations of Leading Word Embeddings Roy Schwartz University of

Pattern-based Solutions to Limitations of Leading Word Embeddings Roy Schwartz University of Washington NLP Seminar, February 8 th , 2016 Joint work with Roi Reichart and Ari Rappoport Background Word embeddings are great! Problem

872 views • 84 slides
Word, Sense and Contextualized Embeddings: Vector Representations of Meaning in NLP Jose

Word, Sense and Contextualized Embeddings: Vector Representations of Meaning in NLP Jose

Word, Sense and Contextualized Embeddings: Vector Representations of Meaning in NLP Jose Camacho-Collados Cardiff University, 18 March 2019 1 Outline Background Vector Space Models (word embeddings) Lexical resources Sense

1.33k views • 98 slides
Symmetric Pattern Based Word Embeddings for Improved Word Similarity Prediction Roy Schwartz + ,

Symmetric Pattern Based Word Embeddings for Improved Word Similarity Prediction Roy Schwartz + ,

Symmetric Pattern Based Word Embeddings for Improved Word Similarity Prediction Roy Schwartz + , Roi Reichart * and Ari Rappoport + + The Hebrew University, * Technion IIT CoNLL 2015 Symmetric Pattern Based Word Embeddings for 2 Improved Word

917 views • 49 slides
Learning Word Embeddings for Low-resource Languages by PU Learning Chao Jiang, Hsiang-Fu Yu,

Learning Word Embeddings for Low-resource Languages by PU Learning Chao Jiang, Hsiang-Fu Yu,

Learning Word Embeddings for Low-resource Languages by PU Learning Chao Jiang, Hsiang-Fu Yu, Cho-Jui Hsieh, Kai-Wei Chang 1 Word Embeddings are useful Many successful stories Named entity recognition Document ranking

697 views • 28 slides
Natural Language Processing Machine Translation Dan Klein UC Berkeley 1 Machine Translation 2

Natural Language Processing Machine Translation Dan Klein UC Berkeley 1 Machine Translation 2

Natural Language Processing Machine Translation Dan Klein UC Berkeley 1 Machine Translation 2 Machine Translation: Examples 3 Levels of Transfer 4 Word Level MT: Examples la politique de la haine . (Foreign Original) politics

675 views • 43 slides
Machine Translation: Examples Statistical NLP Spring 2011 Lecture 7: Phrase-Based MT Dan Klein

Machine Translation: Examples Statistical NLP Spring 2011 Lecture 7: Phrase-Based MT Dan Klein

Machine Translation: Examples Statistical NLP Spring 2011 Lecture 7: Phrase-Based MT Dan Klein UC Berkeley Levels of Transfer World-Level MT: Examples la politique de la haine . (Foreign Original) politics of hate .

250 views • 5 slides
CSE 517 Natural Language Processing Winter 2015 Phrase Based Translation Yejin Choi Slides

CSE 517 Natural Language Processing Winter 2015 Phrase Based Translation Yejin Choi Slides

CSE 517 Natural Language Processing Winter 2015 Phrase Based Translation Yejin Choi Slides from Philipp Koehn, Dan Klein, Luke Zettlemoyer Phrase-Based Systems cat ||| chat ||| 0.9 the cat ||| le chat ||| 0.8 dog ||| chien ||| 0.8 house

755 views • 60 slides
Self-Assembly The spontaneous and reversible association of molecular species to form larger, more

Self-Assembly The spontaneous and reversible association of molecular species to form larger, more

Self-Assembly The spontaneous and reversible association of molecular species to form larger, more complex supramolecular entities according to the intrinsic information contained in the components. Metal-Ligand Interaction Metal as connector : -

834 views • 71 slides
Corpus Acquisition from the Internet Philipp Koehn partially based on slides from Christian Buck

Corpus Acquisition from the Internet Philipp Koehn partially based on slides from Christian Buck

Corpus Acquisition from the Internet Philipp Koehn partially based on slides from Christian Buck 12 November 2020 Philipp Koehn Machine Translation: Corpus Acquisition from the Internet 12 November 2020 Big Data 1 For many language pairs,

594 views • 55 slides
Agenda Part 1: Professional Guest Speakers Spring Recruitment DAS Upcoming Events

Agenda Part 1: Professional Guest Speakers Spring Recruitment DAS Upcoming Events

Agenda Part 1: Professional Guest Speakers Spring Recruitment DAS Upcoming Events Becoming The Ideal Candidate Recruitment Toolkit Part 2: Academic Required Classes Accounting Major CPA Program Options Part 1:

1.08k views • 79 slides
Natural Language Processing Computational Linguistics Text processing Artificial Intelligence

Natural Language Processing Computational Linguistics Text processing Artificial Intelligence

Natural Language Processing Computational Linguistics Text processing Artificial Intelligence Lecture 6 Karim Bouzoubaa Content Acknowledgments Examples Defintions History Objective Levels - Problems Applications

1.03k views • 48 slides
1 Handling Return Traffic Handling Return Traffic URL Switching URL Switching Idea: switch

1 Handling Return Traffic Handling Return Traffic URL Switching URL Switching Idea: switch

The Server Selection Problem The Server Selection Problem server array A server farm B Which server? Server Traffic Management Server Traffic Management not-so-great solutions static client binding manual selection HTTP forwarding Jeff

810 views • 3 slides

More recommend