vector space models
play

Vector Space Models Prof. Sameer Singh CS 295: STATISTICAL NLP - PowerPoint PPT Presentation

Mar 31, 2024 •332 likes •736 views

Vector Space Models Prof. Sameer Singh CS 295: STATISTICAL NLP WINTER 2017 January 19, 2017 Based on slides from Jacob Eisenstein, Noah Smith, Mohit Bansal, Richard Socher, and everyone else they copied from. Outline Latent Semantic Analysis

  1. Vector Space Models Prof. Sameer Singh CS 295: STATISTICAL NLP WINTER 2017 January 19, 2017 Based on slides from Jacob Eisenstein, Noah Smith, Mohit Bansal, Richard Socher, and everyone else they copied from.

  2. Outline Latent Semantic Analysis Vector Models for Words Reducing the Dimensions Direct Embeddings CS 295: STATISTICAL NLP (WINTER 2017) 2

  3. Outline Latent Semantic Analysis Vector Models for Words Reducing the Dimensions Direct Embeddings CS 295: STATISTICAL NLP (WINTER 2017) 3

  4. Example: Documents c1: Human machine interface for ABC computer applications c2: A survey of user opinion of computer system response time c3: The EPS user interface management system c4: System and human system engineering testing of EPS c5: Relation of user perceived response time to error measurement m1: The generation of random, binary, ordered trees m2: The intersection graph of paths in trees m3: Graph minors IV: Widths of trees and well-quasi-ordering m4: Graph minors: A survey From http://lsa.colorado.edu/papers/dp1.LSAintro.pdf CS 295: STATISTICAL NLP (WINTER 2017) 4

  5. Example: Term-Doc Matrix c1 c2 c3 c4 c5 m1 m2 m3 m4 human interface computer user system response time EPS survey trees graph minors CS 295: STATISTICAL NLP (WINTER 2017) 5

  6. Problems with Sparse Vectors c2: A survey of user opinion of computer system response time c1: Human machine interface m4: Graph minors: A survey for ABC computer applications CS 295: STATISTICAL NLP (WINTER 2017) 6

  7. Example: Distance Matrix c1 c2 c3 c4 c5 m1 m2 m3 m4 c1 c2 c3 c4 c5 m1 m2 m3 m4 CS 295: STATISTICAL NLP (WINTER 2017) 7

  8. Option 2: SVD CS 295: STATISTICAL NLP (WINTER 2017) 8

  9. Latent Semantic Analysis (LSA) CS 295: STATISTICAL NLP (WINTER 2017) 9

  10. Example: Decomposition c1 c2 c3 c4 c5 m1 m2 m3 m4 human interface computer user system response time EPS survey trees graph minors CS 295: STATISTICAL NLP (WINTER 2017) 10

  11. New Document Vectors CS 295: STATISTICAL NLP (WINTER 2017) 11

  12. Example: Reconstruction human interface computer user system response time EPS survey trees graph minors CS 295: STATISTICAL NLP (WINTER 2017) 12

  13. Example: Distance Matrix CS 295: STATISTICAL NLP (WINTER 2017) 13

  14. Outline Latent Semantic Analysis Vector Models for Words Reducing the Dimensions Direct Embeddings CS 295: STATISTICAL NLP (WINTER 2017) 14

  15. Let’s look at words A bottle of tezguino is on the table. Everybody likes tezguino. Tezguino makes you drunk. We make tezguino out of corn. What does tezguino mean? Loud, motor oil, tortillas, choices, wine You shall know a word by the company keeps. (Firth, 1957) CS 295: STATISTICAL NLP (WINTER 2017) 15

  16. Term-Context Matrix C1 C2 C3 C4 tezguino C1: A bottle of ______ is on the table. loud C2: Everybody likes ______. motor oil C3: _____ makes you drunk. tortillas C4: We make _____ out of corn. choices wine CS 295: STATISTICAL NLP (WINTER 2017) 16

  17. What is a “Context”? A bottle of tezguino is on the table. Tezguino makes you drunk. Can be anything you want! … • Entire contents of the sentence I had a fancy bottle of wine and One word before and after • got drunk last night! • Words in the same sentence The terrible wine is on the table. Document it appears in • Many other variations… • CS 295: STATISTICAL NLP (WINTER 2017) 17

  18. What is a “Context”? A bottle of tezguino is on the table. Tezguino makes you drunk. Can be anything you want! … • Entire contents of the sentence I had a fancy bottle of wine and Unlikely to occur again! • got drunk last night! • One word before and after The terrible wine is on the table. Words in the same sentence • Document ID it appears in • • Many other variations… C1 C2 C3 C4 tezguino wine CS 295: STATISTICAL NLP (WINTER 2017) 18

  19. What is a “Context”? A bottle of tezguino is on the table. Tezguino makes you drunk. Can be anything you want! … • Entire contents of the sentence I had a fancy bottle of wine and One word before and after • got drunk last night! • Or n-words The terrible wine is on the table. Words in the same sentence • Document it appears in • • Many other variations… bottle-of is-of makes-you and-got the-terrible is-on tezguino wine CS 295: STATISTICAL NLP (WINTER 2017) 19

  20. What is a “Context”? A bottle of tezguino is on the table. Tezguino makes you drunk. Can be anything you want! … • Entire contents of the sentence I had a fancy bottle of wine and One word before and after • got drunk last night! • Words in the same sentence The terrible wine is on the table. Filter: nouns and verbs? • Bag of words in a window • • Document it appears in Many other variations… • bottle table you drunk fancy night terrible tezguino wine CS 295: STATISTICAL NLP (WINTER 2017) 20

  21. What is a “Context”? A bottle of tezguino is on the table. Tezguino makes you drunk. Can be anything you want! … • Entire contents of the sentence I had a fancy bottle of wine and One word before and after • got drunk last night! • Words in the same sentence The terrible wine is on the table. Document it appears in • Term-document matrix! • D1 D2 D3 D4 • Latent Semantic Analysis tezguino Many other variations… • table bottle drunk wine CS 295: STATISTICAL NLP (WINTER 2017) 21

  22. Pointwise Mutual Information Raw counts are not good Skewed towards common words/contexts • Many of them are not informative • is, the, it, they, … • PMI(w,c) How much more likely is w to occur in c, than just randomly? • CS 295: STATISTICAL NLP (WINTER 2017) 22

  23. Outline Latent Semantic Analysis Vector Models for Words Reducing the Dimensions Direct Embeddings CS 295: STATISTICAL NLP (WINTER 2017) 23

  24. Option 1: Revisiting Clustering CS 295: STATISTICAL NLP (WINTER 2017) 24

  25. Hierarchical Clustering CS 295: STATISTICAL NLP (WINTER 2017) 25

  26. Example CS 295: STATISTICAL NLP (WINTER 2017) 26

  27. Brown Clusters for Twitter http://www.cs.cmu.edu/~ark/TweetNLP/cluster_viewer.html CS 295: STATISTICAL NLP (WINTER 2017) 27

  28. Option 2: SVD CS 295: STATISTICAL NLP (WINTER 2017) 28

  29. Example Word Projection CS 295: STATISTICAL NLP (WINTER 2017) 29

  30. Problem with SVD & Clustering Computational Complexity SVD: O(mn 2 ) • Clustering: O(knm) per iteration, or O(n 3 ) • But, n can be 100,000! • “One shot” Difficult to add new documents or words • Cannot work with streaming data • CS 295: STATISTICAL NLP (WINTER 2017) 30

  31. Outline Latent Semantic Analysis Vector Models for Words Reducing the Dimensions Direct Embeddings CS 295: STATISTICAL NLP (WINTER 2017) 31

  32. Predict surrounding words A bottle of tezguino is on the table. u v CS 295: STATISTICAL NLP (WINTER 2017) 32

  33. Estimating the Word Vectors CS 295: STATISTICAL NLP (WINTER 2017) 33

  34. Similar Meaning = Close CS 295: STATISTICAL NLP (WINTER 2017) 34

  35. Similar Meaning = Close https://siddhant7.github.io/Vector-Representation-of-Words/ CS 295: STATISTICAL NLP (WINTER 2017) 35

  36. Vectors “know” Gender male : female :: King : queen King - male + female queen https://siddhant7.github.io/Vector-Representation-of-Words/ CS 295: STATISTICAL NLP (WINTER 2017) 36

  37. They “know” Tenses! walking : walked :: swimming : swam swimming – walking + walked swam https://siddhant7.github.io/Vector-Representation-of-Words/ CS 295: STATISTICAL NLP (WINTER 2017) 37

  38. They “know” Facts! Madrid Country – Capital + Spain https://siddhant7.github.io/Vector-Representation-of-Words/ CS 295: STATISTICAL NLP (WINTER 2017) 38

  39. Upcoming… • Homework 1 is up! No more material will be covered Homework • • Due: January 26, 2017 Project pitch is due January 23, 2017! • Start assembling teams now Project • Tons of datasets on the “projects” page on website • CS 295: STATISTICAL NLP (WINTER 2017) 39

Download Presentation
Download Policy: The content available on the website is offered to you 'AS IS' for your personal information and use only. It cannot be commercialized, licensed, or distributed on other websites without prior consent from the author. To download a presentation, simply click this link. If you encounter any difficulties during the download process, it's possible that the publisher has removed the file from their server.

Recommend

Boolean and Vector Space Retrieval Models CS 290N Some of slides from R. Mooney (UTexas),

Boolean and Vector Space Retrieval Models CS 290N Some of slides from R. Mooney (UTexas),

Boolean and Vector Space Retrieval Models CS 290N Some of slides from R. Mooney (UTexas), J. Ghosh (UT ECE), D. Lee (USTHK). 1 Table of Content Which results satisfy the query constraint? Boolean model Statistical vector space

570 views • 36 slides
Vector Space Models Module Introduction CS6200: Information Retrieval In the first module, we

Vector Space Models Module Introduction CS6200: Information Retrieval In the first module, we

Vector Space Models Module Introduction CS6200: Information Retrieval In the first module, we introduced Vector Space Models as an alternative to Boolean Retrieval. This module discusses VSMs in a lot more detail. By the end of the module, you

252 views • 21 slides
Efficient Estimation of Word Representation in Vector Space Topics Language Models in NLP o

Efficient Estimation of Word Representation in Vector Space Topics Language Models in NLP o

Efficient Estimation of Word Representation in Vector Space Topics Language Models in NLP o Markov Models (n-gram model) o Distributed Representation of words o Motivation for word vector model of data o Feedforward Neural Network

308 views • 27 slides
Boolean and Vector Space Retrieval Models CS 293S, 2017 Some of slides from R. Mooney

Boolean and Vector Space Retrieval Models CS 293S, 2017 Some of slides from R. Mooney

Boolean and Vector Space Retrieval Models CS 293S, 2017 Some of slides from R. Mooney (UTexas), J. Ghosh (UT ECE), D. Lee (USTHK). 1 Table of Content Which results satisfy the query constraint? Boolean model Statistical

473 views • 31 slides
Natural Language Processing and Information Retrieval Indexing and Vector Space Models

Natural Language Processing and Information Retrieval Indexing and Vector Space Models

Natural Language Processing and Information Retrieval Indexing and Vector Space Models Alessandro Moschitti Department of Computer Science and Information Engineering University of Trento Email: moschitti@disi.unitn.it Outline Preprocessing

871 views • 39 slides
Natural Language Processing and Information Retrieval Indexing and Vector Space Models

Natural Language Processing and Information Retrieval Indexing and Vector Space Models

Natural Language Processing and Information Retrieval Indexing and Vector Space Models Alessandro Moschitti Department of Computer Science and Information Engineering University of Trento Email: moschitti@disi.unitn.it Lastlecture

554 views • 54 slides
Vector-space models of meaning Christopher Potts CS 244U: Natural language understanding Jan 19

Vector-space models of meaning Christopher Potts CS 244U: Natural language understanding Jan 19

Overview Matrix designs Weighting/normalization Distance measures Experiments Dimensionality reduction Tools Looking ahead Vector-space models of meaning Christopher Potts CS 244U: Natural language understanding Jan 19 1 / 48 Overview

789 views • 65 slides
27. Vector fields in space A vector field in space is given by + R F = P + Q

27. Vector fields in space A vector field in space is given by + R F = P + Q

27. Vector fields in space A vector field in space is given by + R F = P + Q k = P, Q, R . Here the components, P , Q and R are scalar functions of x , y and z . F could be a force field; F = c x, y, z

310 views • 3 slides
Vector Space Models for Phrase-based Machine Translation Tamer Alkhouli, Andreas Guta, and

Vector Space Models for Phrase-based Machine Translation Tamer Alkhouli, Andreas Guta, and

Vector Space Models for Phrase-based Machine Translation Tamer Alkhouli, Andreas Guta, and Hermann Ney <surname>@cs.rwth-aachen.de Eighth Workshop on Syntax, Semantics and Structure in Statistical Translation Doha, Qatar October 25, 2014

275 views • 16 slides
Introduction to Natural Language Processing Evaluation Vector space model Term weighting Ranked

Introduction to Natural Language Processing Evaluation Vector space model Term weighting Ranked

Introduction Week 7, lecture WWW: pecina@ufal.mfg.cuni.cz E-mail: Pavel Pecina Todays teacher: Boolen and Vector Space Models for Information Retrieval Todays topic: Today: Boolean retrieval by members of the Institute of Formal and

1.45k views • 98 slides
Syntax versus Semantics: Analysis of Enriched Vector Space Models Benno Stein and Sven Meyer zu

Syntax versus Semantics: Analysis of Enriched Vector Space Models Benno Stein and Sven Meyer zu

Syntax versus Semantics: Analysis of Enriched Vector Space Models Benno Stein and Sven Meyer zu Eissen and Martin Potthast Bauhaus University Weimar Introduction Enrichment Approaches Evaluation TIR06 Aug. 29th, 2006 Stein/Meyer zu

336 views • 33 slides
The Classic Vector Space Model Description, Advantages and Limitations of the Classic Vector

The Classic Vector Space Model Description, Advantages and Limitations of the Classic Vector

The Classic Vector Space Model Description, Advantages and Limitations of the Classic Vector Space Model Dr. E. Garcia Global Information Unlike the Term Count Model, Salton's Vector Space Model [1] incorporates local and global information Eq

477 views • 8 slides
Overview Last week we introduced the notion of an abstract vector space, and we saw that

Overview Last week we introduced the notion of an abstract vector space, and we saw that

Overview Last week we introduced the notion of an abstract vector space, and we saw that apparently different sets like polynomials, continuous functions, and symmetric matrices all satisfy the 10 axioms defining a vector space. We also discussed

731 views • 31 slides
Overview Given two bases B and C for the same vector space, we saw yesterday how P P to find the

Overview Given two bases B and C for the same vector space, we saw yesterday how P P to find the

Overview Given two bases B and C for the same vector space, we saw yesterday how P P to find the change of coordinates matrices CB nd BC . Such a matrix is always square, since every basis for a vector space V has the same number of

832 views • 61 slides
Part 10: Vector Space Classification Francesco Ricci 1 Content p Recap on nave Bayes p

Part 10: Vector Space Classification Francesco Ricci 1 Content p Recap on nave Bayes p

Part 10: Vector Space Classification Francesco Ricci 1 Content p Recap on nave Bayes p Vector space methods for Text Classification n K Nearest Neighbors p Bayes error rate n Decision boundaries n Vector space classification

1.09k views • 95 slides
Models for Models for Retrieval and Browsing Retrieval and Browsing - Fuzzy Set, Extended

Models for Models for Retrieval and Browsing Retrieval and Browsing - Fuzzy Set, Extended

Models for Models for Retrieval and Browsing Retrieval and Browsing - Fuzzy Set, Extended Boolean, Generalized Vector Space Models Berlin Chen 2004 Reference: 1. Modern Information Retrieval . Chapter 2 Taxonomy of Classic IR Models Set

511 views • 30 slides
The Dialectica Translation of Type Theory Andrej Bauer Pierre-Marie Pdrot University of

The Dialectica Translation of Type Theory Andrej Bauer Pierre-Marie Pdrot University of

The Dialectica Translation of Type Theory Andrej Bauer Pierre-Marie Pdrot University of Ljubljana INRIA TYPES 24th May 2016 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

642 views • 50 slides
Slice knots which bound Klein bottles Arunima Ray AMS Central Sectional Meeting University of

Slice knots which bound Klein bottles Arunima Ray AMS Central Sectional Meeting University of

Goal Introduction Preliminaries Main theorem Interlude Corollaries Slice knots which bound Klein bottles Arunima Ray AMS Central Sectional Meeting University of Akron Akron, Ohio October 20, 2012 Goal Introduction Preliminaries Main

395 views • 36 slides
ENGINEERING Environmental Engineering Module 5.2 Proudly developed by SMART with funding from

ENGINEERING Environmental Engineering Module 5.2 Proudly developed by SMART with funding from

ENGINEERING Environmental Engineering Module 5.2 Proudly developed by SMART with funding from Inspiring Australia Environmental Engineers Video: What is Environmental Engineering?

363 views • 14 slides
Dense Random Fields Philipp Krhenbhl Stanford University Zoo of computer vision problems

Dense Random Fields Philipp Krhenbhl Stanford University Zoo of computer vision problems

Dense Random Fields Philipp Krhenbhl Stanford University Zoo of computer vision problems bottle tiger bottle bottle kitten wood car skin paper pumpkin cloth playing tennis Emma in her hat looking super cute 2 Zoo of computer

1.61k views • 143 slides
Ranger zone (multi-step problems) DATE LO: To complete a post assessment on volume and capacity

Ranger zone (multi-step problems) DATE LO: To complete a post assessment on volume and capacity

Maths Warm Up:- Education City Problem solving Party Problems (problems involving measurement and all 4 operations) Ranger zone (multi-step problems) DATE LO: To complete a post assessment on volume and capacity We have now finished our

267 views • 8 slides
Commonsense in Natural Language Yonatan Bisk , Rowan Zellers, Ronan Le Bras, Jianfeng Gao and Yejin

Commonsense in Natural Language Yonatan Bisk , Rowan Zellers, Ronan Le Bras, Jianfeng Gao and Yejin

PIQA: Reasoning about Physical Commonsense in Natural Language Yonatan Bisk , Rowan Zellers, Ronan Le Bras, Jianfeng Gao and Yejin Choi Whats the meaning of a word? 01 ) This is silly right?

301 views • 19 slides
Experiment in a bottle (magnetic): MICE PRY Holger Witte Brookhaven National Laboratory

Experiment in a bottle (magnetic): MICE PRY Holger Witte Brookhaven National Laboratory

Experiment in a bottle (magnetic): MICE PRY Holger Witte Brookhaven National Laboratory Advanced Accelerator Group 21 May 2015 1 Overview Introduction and Concept CAM concept (Forces on PRY) Status of Manufacturing Effect of

976 views • 40 slides
nd Conce e 2 nd The e cont ntext ext : The ncern rn for r innov ovati ation on - Case e

nd Conce e 2 nd The e cont ntext ext : The ncern rn for r innov ovati ation on - Case e

nd Conce e 2 nd The e cont ntext ext : The ncern rn for r innov ovati ation on - Case e study dy on Filter ter Bottle tle for r CRPF F DE 402 Introduction to Design Case Studies ies 1st concern for innovation The Cause

979 views • 64 slides

More recommend