learning word representations by jointly modeling
play

Learning Word Representations by Jointly Modeling Syntagmatic and - PowerPoint PPT Presentation

Oct 21, 2022 •117 likes •524 views

Learning Word Representations by Jointly Modeling Syntagmatic and Paradigmatic Relations Fei Sun , Jiafeng Guo, Yanyan Lan, Jun Xu, and Xueqi Cheng CAS Key Lab of Network Data Science and Technology Institute of Computing Technology, Chinese

  1. Learning Word Representations by Jointly Modeling Syntagmatic and Paradigmatic Relations Fei Sun , Jiafeng Guo, Yanyan Lan, Jun Xu, and Xueqi Cheng CAS Key Lab of Network Data Science and Technology Institute of Computing Technology, Chinese Academy of Sciences July 20, 2015 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Fei Sun W ORD R EPRESENTATION July 20, 2015 1 / 28

  2. Word Representations Sentiment Machine Analysis Translation [Kalchbrenner and [Maas et al., Blunsom, 2013] 2011] language POS Word Taging modeling Representations [Collobert et al., [Bengio et al., 2011] 2003] Word-Sense Parsing Disambiguation [Socher et al., [Collobert et al., 2011] 2011] . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Fei Sun W ORD R EPRESENTATION July 20, 2015 2 / 28

  3. Word Representations Models GloVe [Pennington et al., 2014] Word2Vec [Mikolov et al., 2013] LBL [Mnih and Hinton, 2007] NPLMs [Bengio et al., 2003] LDA [Blei et al., 2003] HAL [Deerwester et al., 1990] LSI BOW [Lund et al., 1995] [Harris, 1954] 1940 1960 1980 2000 2020 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Fei Sun W ORD R EPRESENTATION July 20, 2015 3 / 28

  4. Word Representations Models GloVe [Pennington et al., 2014] Word2Vec [Mikolov et al., 2013] LBL [Mnih and Hinton, 2007] NPLMs [Bengio et al., 2003] Relations? LDA [Blei et al., 2003] HAL [Deerwester et al., 1990] LSI BOW [Lund et al., 1995] [Harris, 1954] 1940 1960 1980 2000 2020 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Fei Sun W ORD R EPRESENTATION July 20, 2015 3 / 28

  5. Relations One Hypothesis Two Interpretation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Fei Sun W ORD R EPRESENTATION July 20, 2015 4 / 28

  6. The Distributional Hypothesis [Harris, 1954, Firth, 1957] “ You shall know a word by the company it keeps .” —J.R. Firth . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Fei Sun W ORD R EPRESENTATION July 20, 2015 5 / 28

  7. Syntagmatic and Paradigmatic Relations [Gabrilovich and Markovitch, 2007] syntagmatic was a physicist. Albert Einstein paradigmatic Feynman was a physicist. Richard syntagmatic • Syntagmatic: words co-occur in the same text region • Paradigmatic: words occur in the same context, may not at the same time . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Fei Sun W ORD R EPRESENTATION July 20, 2015 6 / 28

  8. Syntagmatic syntagmatic was a physicist. Albert Einstein Feynman was a physicist. Richard syntagmatic . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Fei Sun W ORD R EPRESENTATION July 20, 2015 7 / 28

  9. Syntagmatic syntagmatic was a physicist. Albert Einstein Feynman was a physicist. Richard syntagmatic d 1 d 2 Einstein 1 0 Feynman 0 1 physicist 1 1 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Fei Sun W ORD R EPRESENTATION July 20, 2015 7 / 28

  10. Syntagmatic syntagmatic was a physicist. Albert Einstein Feynman was a physicist. Richard syntagmatic Feynman physicist 1 d 1 d 2 Einstein 1 0 Feynman 0 1 physicist 1 1 Einstein 0 1 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Fei Sun W ORD R EPRESENTATION July 20, 2015 7 / 28

  11. Syntagmatic syntagmatic was a physicist. Albert Einstein Feynman was a physicist. Richard syntagmatic Feynman physicist 1 d 1 d 2 Einstein 1 0 Feynman 0 1 physicist 1 1 Einstein 0 1 LSI, LDA, PV-DBOW · · · . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Fei Sun W ORD R EPRESENTATION July 20, 2015 7 / 28

  12. Paradigmatic was a physicist. Albert Einstein paradigmatic Feynman was a physicist. Richard . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Fei Sun W ORD R EPRESENTATION July 20, 2015 8 / 28

  13. Paradigmatic was a physicist. Albert Einstein paradigmatic Feynman was a physicist. Richard Einstein Feynman physicist Einstein 0 0 1 Feynman 0 0 1 physicist 1 1 0 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Fei Sun W ORD R EPRESENTATION July 20, 2015 8 / 28

  14. Paradigmatic was a physicist. Albert Einstein paradigmatic Feynman was a physicist. Richard Einstein Feynman physicist Einstein 0 0 1 Einstein Feynman Feynman 0 0 1 physicist physicist 1 1 0 0 0 0 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Fei Sun W ORD R EPRESENTATION July 20, 2015 8 / 28

  15. Paradigmatic was a physicist. Albert Einstein paradigmatic Feynman was a physicist. Richard Einstein Feynman physicist Einstein 0 0 1 Einstein Feynman Feynman 0 0 1 physicist physicist 1 1 0 0 0 0 NLMs, Word2Vec, GloVe · · · . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Fei Sun W ORD R EPRESENTATION July 20, 2015 8 / 28

  16. Motivation syntagmatic was a physicist. Albert Einstein paradigmatic Word2Vec PV-DBOW Feynman was a physicist. Richard syntagmatic . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Fei Sun W ORD R EPRESENTATION July 20, 2015 9 / 28

  17. Model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Fei Sun W ORD R EPRESENTATION July 20, 2015 10 / 28

  18. nw nw Parallel Document Context Model (PDC) . . . N ∑ ∑ log p ( w n i | h n ℓ = i ) the w n n =1 i ∈ d n Paradigmatic cat on the sat . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Fei Sun W ORD R EPRESENTATION July 20, 2015 11 / 28

  19. nw nw Parallel Document Context Model (PDC) . . . N ( ) ∑ ∑ log p ( w n i | h n i )+ log p ( w n ℓ = i | d n ) the w n n =1 i ∈ d n Paradigmatic cat on the sat . . . . . . the cat sat Syntagmatic on the . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Fei Sun W ORD R EPRESENTATION July 20, 2015 11 / 28

  20. Parallel Document Context Model (PDC) . . . N ( ) ∑ ∑ log p ( w n i | h n i )+ log p ( w n ℓ = i | d n ) the w n n =1 i ∈ d n Paradigmatic Negative Sampling cat N ( ∑ ∑ log σ ( ⃗ i · ⃗ i )+ log σ ( ⃗ i · ⃗ w n h n w n ℓ = d n ) on w n n =1 i ∈ d n w ′ · ⃗ + k · E w ′ ∼ P nw log σ ( ⃗ h n i ) the ) w ′ · ⃗ sat + k · E w ′ ∼ P nw log σ ( ⃗ d n ) . 1 . σ ( x ) = . 1 + exp ( − x ) . . . the cat sat Syntagmatic on the . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Fei Sun W ORD R EPRESENTATION July 20, 2015 11 / 28

  21. Parallel Document Context Model (PDC) . . . N ( ) ∑ ∑ log p ( w n i | h n i )+ log p ( w n ℓ = i | d n ) the w n n =1 i ∈ d n Paradigmatic Negative Sampling cat N ( ∑ ∑ log σ ( ⃗ i · ⃗ i )+ log σ ( ⃗ i · ⃗ w n h n w n ℓ = d n ) on w n n =1 i ∈ d n w ′ · ⃗ + k · E w ′ ∼ P nw log σ ( ⃗ h n i ) the ) w ′ · ⃗ sat + k · E w ′ ∼ P nw log σ ( ⃗ d n ) . 1 . σ ( x ) = . 1 + exp ( − x ) . . . the cat sat PDC PV-DM Syntagmatic on the . . . MF for W-D + W-C not clear . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Fei Sun W ORD R EPRESENTATION July 20, 2015 11 / 28

  22. nw nc Hierarchical Document Context Model (HDC) N ∑ ∑ log p ( w n ℓ = i | d n ) w n n =1 i ∈ d n sat Syntagmatic . . . the cat sat on the . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Fei Sun W ORD R EPRESENTATION July 20, 2015 12 / 28

  23. nw nc Hierarchical Document Context Model (HDC) on cat · · · the the · · · N i + L ( ) ∑ ∑ ∑ log p ( w n log p ( c n j | w n ℓ = i | d n )+ i ) w n n =1 i ∈ d n j = i − L j ̸ = i Paradigmatic sat Syntagmatic . . . the cat sat on the . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Fei Sun W ORD R EPRESENTATION July 20, 2015 12 / 28

Download Presentation
Download Policy: The content available on the website is offered to you 'AS IS' for your personal information and use only. It cannot be commercialized, licensed, or distributed on other websites without prior consent from the author. To download a presentation, simply click this link. If you encounter any difficulties during the download process, it's possible that the publisher has removed the file from their server.

Recommend

Learning Word Representations by Jointly Modeling Syntagmatic and Paradigmatic Relations Fei Sun ,

Learning Word Representations by Jointly Modeling Syntagmatic and Paradigmatic Relations Fei Sun ,

Learning Word Representations by Jointly Modeling Syntagmatic and Paradigmatic Relations Fei Sun , Jiafeng Guo, Yanyan Lan, Jun Xu, and Xueqi Cheng ofey.sunfei@gmail.com CAS Key Lab of Network Data Science and Technology Institute of Computing

1.02k views • 41 slides
Jointly Learning Word Representations and Composition Functions Using Predicate-Argument

Jointly Learning Word Representations and Composition Functions Using Predicate-Argument

Jointly Learning Word Representations and Composition Functions Using Predicate-Argument Structures Kazuma Hashimoto (UT) Pontus Stenetorp (UT) Makoto Miwa (TTI) Yoshimasa Tsuruoka (UT) U niversity of T okyo ( UT ) T oyota T echnological I

1.39k views • 119 slides
Inside Out: Two Jointly Predictive Models for Word Representations and Phrase Representations Fei

Inside Out: Two Jointly Predictive Models for Word Representations and Phrase Representations Fei

Inside Out: Two Jointly Predictive Models for Word Representations and Phrase Representations Fei Sun , Jiafeng Guo, Yanyan Lan, Jun Xu, and Xueqi Cheng ofey.sunfei@gmail.com CAS Key Lab of Network Data Science and Technology Institute of

701 views • 37 slides
Jointly Learning Word and Phrase Embeddings Using Neural Networks and Implicit Tensor

Jointly Learning Word and Phrase Embeddings Using Neural Networks and Implicit Tensor

Jointly Learning Word and Phrase Embeddings Using Neural Networks and Implicit Tensor Factorization Kazuma Hashimoto Tsuruoka Laboratory, University of Tokyo 19/06/2015 Talk@UCL Machine Reading Lab. Self Introduction Name Kazuma

726 views • 39 slides
NLU lecture 5: Word representations and morphology Adam Lopez alopez@inf.ed.ac.uk Essential

NLU lecture 5: Word representations and morphology Adam Lopez alopez@inf.ed.ac.uk Essential

NLU lecture 5: Word representations and morphology Adam Lopez alopez@inf.ed.ac.uk Essential epistemology Word representations and word2vec Word representations and compositional morphology Reading: Mikolov et al. 2013, Luong et al.

746 views • 42 slides
Incorporating Relational Knowledge into Word Representations using Subspace Regularization Jun

Incorporating Relational Knowledge into Word Representations using Subspace Regularization Jun

Incorporating Relational Knowledge into Word Representations using Subspace Regularization Jun Araki (Carnegie Mellon University) joint work with Abhishek Kumar (IBM Research) ACL 2016 Distributed word representations Low-dimensional dense

378 views • 18 slides
Contextual Word Representations with BERT and Other Pre-trained Language Models Jacob Devlin

Contextual Word Representations with BERT and Other Pre-trained Language Models Jacob Devlin

Contextual Word Representations with BERT and Other Pre-trained Language Models Jacob Devlin Google AI Language History and Background Pre-training in NLP Word embeddings are the basis of deep learning for NLP king queen [-0.5, -0.9,

852 views • 43 slides
Batch IS NOT Heavy: Learning Word Representations From All Samples 1 1 1 Xin Xin, Fajie Yuan,

Batch IS NOT Heavy: Learning Word Representations From All Samples 1 1 1 Xin Xin, Fajie Yuan,

Batch IS NOT Heavy: Learning Word Representations From All Samples 1 1 1 Xin Xin, Fajie Yuan, Xiangnan He, Joemon Jose 2 1 School of Computing Science, University of Glasgow 2 School of Computing, National University of Singapore Presented by

603 views • 28 slides
Word representations Benoit Favre < benoit.favre@univ-mrs.fr > Aix-Marseille Universit,

Word representations Benoit Favre < benoit.favre@univ-mrs.fr > Aix-Marseille Universit,

Deep learning for natural language processing Word representations Benoit Favre < benoit.favre@univ-mrs.fr > Aix-Marseille Universit, LIF/CNRS 21 Feb 2017 Benoit Favre (AMU) DL4NLP: word embeddings 21 Feb 2017 1 / 26 Deep learning

432 views • 26 slides
Word Meaning: Distributional Representations & Word Sense Disambiguation CMSC 723 / LING 723

Word Meaning: Distributional Representations & Word Sense Disambiguation CMSC 723 / LING 723

Word Meaning: Distributional Representations & Word Sense Disambiguation CMSC 723 / LING 723 / INST 725 Marine Carpuat Slides credit: Dan Jurafsky Reminders Read the syllabus Make sure you have access to piazza Get started on

691 views • 54 slides
Learning Representations for Multi-Dimensional Sentiment Andrew Maas, Peter Pham, Ying Wang Word

Learning Representations for Multi-Dimensional Sentiment Andrew Maas, Peter Pham, Ying Wang Word

Learning Representations for Multi-Dimensional Sentiment Andrew Maas, Peter Pham, Ying Wang Word Vectors Maas et al (ACL 2011) The Experience Project this is sad, i just roasted marshmallow. by myself. on the stove top. lol. at least they

72 views • 6 slides
Sentence and Contextualised Word Representations Graham Neubig Site

Sentence and Contextualised Word Representations Graham Neubig Site

CS11-747 Neural Networks for NLP Sentence and Contextualised Word Representations Graham Neubig Site https://phontron.com/class/nn4nlp2019/ (w/ slides by Antonis Anastasopoulos) Sentence Representations We can create a vector or sequence

619 views • 43 slides
Lesson 10 Deep learning for NLP: Mul6lingual Word Sequence Modeling December 15, 2016 EPFL

Lesson 10 Deep learning for NLP: Mul6lingual Word Sequence Modeling December 15, 2016 EPFL

Human Language Technology: Applica6on to Informa6on Access Lesson 10 Deep learning for NLP: Mul6lingual Word Sequence Modeling December 15, 2016 EPFL Doctoral Course EE-724 Nikolaos Pappas Idiap Research Ins6tute, Mar6gny Outline of the

1.6k views • 86 slides
Using Machine Learning to Study the Neural Representations of Language Meanings Tom M. Mitchell

Using Machine Learning to Study the Neural Representations of Language Meanings Tom M. Mitchell

Using Machine Learning to Study the Neural Representations of Language Meanings Tom M. Mitchell Carnegie Mellon University June 2017 How does neural activity encode word meanings? How does neural activity encode word meanings? How does

673 views • 64 slides
Representations of Power Series over Word Algebras Ulrich Faigle, Alexander Schnhuth

Representations of Power Series over Word Algebras Ulrich Faigle, Alexander Schnhuth

Guideline Introduction Representations Solutions Conclusion Representations of Power Series over Word Algebras Ulrich Faigle, Alexander Schnhuth Mathematical Institute University Cologne Centrum Wiskunde & Informatica Amsterdam CTW

534 views • 39 slides
Word representations and modelling ambiguity: A case study of metaphor Ekaterina Shutova ILLC

Word representations and modelling ambiguity: A case study of metaphor Ekaterina Shutova ILLC

Word representations and modelling ambiguity: A case study of metaphor Ekaterina Shutova ILLC University of Amsterdam 2 May 2018 Word representations and modelling ambiguity: A case study of metaphor 1 / 29 Polysemy and word senses The

790 views • 29 slides
Agenda Agenda Linda Rammler, UConn UCEDD (copy from Agenda handout) Fr. John Gallagher,

Agenda Agenda Linda Rammler, UConn UCEDD (copy from Agenda handout) Fr. John Gallagher,

2/21/18 Agenda Agenda Linda Rammler, UConn UCEDD (copy from Agenda handout) Fr. John Gallagher, O.F.M. Cap. (St. Pius X Church) Kim Ranell Newgass (Baptist Church in New Haven) Mary Lou Freitas (Bethany Covenant) Doris

508 views • 5 slides
The Privacy of Secured Computations Adam Smith Penn State Crypto & Big Data Workshop

The Privacy of Secured Computations Adam Smith Penn State Crypto & Big Data Workshop

The Privacy of Secured Computations Adam Smith Penn State Crypto & Big Data Workshop Relax it can only December 15, 2015 see metadata. Cartoon: 1 Big Data Every <length of time> your <household object> generates

832 views • 55 slides
UL HPC School 2017 Overview & Challenges of the UL HPC Facility at the Belval & EuroHPC

UL HPC School 2017 Overview & Challenges of the UL HPC Facility at the Belval & EuroHPC

UL HPC School 2017 Overview & Challenges of the UL HPC Facility at the Belval & EuroHPC Horizon Prof. Pascal Bouvry, Dr. Sebastien Varrette and the UL HPC Team June. 12 th , 2017, MSA 4.510 University of Luxembourg (UL), Luxembourg

2.13k views • 174 slides
Unimodality of q -Eulerian polynomials and q , p -Eulerian polynomials Michelle Wachs University

Unimodality of q -Eulerian polynomials and q , p -Eulerian polynomials Michelle Wachs University

Unimodality of q -Eulerian polynomials and q , p -Eulerian polynomials Michelle Wachs University of Miami Joint work with John Shareshian and with Anthony Henderson Eulerian numbers and Eulerian polynomials Eulerian numbers: a n , j := |{

710 views • 43 slides
[ with the schedulers collaboration] Alberto Miranda, PhD Researcher on HPC I/O

[ with the schedulers collaboration] Alberto Miranda, PhD Researcher on HPC I/O

www.bsc.es echofs: Enabling Transparent Access to Node-local NVM Burst Buffers for Legacy Applications [ with the schedulers collaboration] Alberto Miranda, PhD Researcher on HPC I/O alberto.miranda@bsc.es Dagstuhl, May 2017 The

768 views • 22 slides
Highlights from BESIII Hai-Bo Li for BESIII Collaboration Institute of High Energy Physics

Highlights from BESIII Hai-Bo Li for BESIII Collaboration Institute of High Energy Physics

Highlights from BESIII Hai-Bo Li for BESIII Collaboration Institute of High Energy Physics Beijing, China XIV International Conference on Hadron Spectroscopy June 13 17, 2011 Knstlerhaus, in Mnchen , Germany June 15, 2011 Hai-Bo Li

1.09k views • 54 slides
Re Recognizing Afffect in Dialog Systems ms Nate Perkins

Re Recognizing Afffect in Dialog Systems ms Nate Perkins

Re Recognizing Afffect in Dialog Systems ms Nate Perkins Problem Iden.fy emo.onal state in human speech dialog Why? Tutoring systems Call center

1.01k views • 71 slides
Linear Programming in Low Dimensions (most slides by Nati Srebro) September 11, 2003 Lecture 3:

Linear Programming in Low Dimensions (most slides by Nati Srebro) September 11, 2003 Lecture 3:

Linear Programming in Low Dimensions (most slides by Nati Srebro) September 11, 2003 Lecture 3: Linear Programming in Low Dimensions Linear Programming + + + Maximize : c x c x c x 1 1 2 2 d d + + + Subject to : a

772 views • 43 slides

More recommend