Language and Stats 11-(7/6)61 Heterogenity of language Types and - PowerPoint PPT Presentation

Heterogeneity • The Dover mail was in its usual genial position that the guard suspected the passengers, the passengers suspected one another and the guard, they all suspected everybody else, and the coachman was sure of nothing but the horses; as to which cattle he could with a clear conscience have taken his oath on the two Testaments that they were not fit for the journey. • News or fiction? • Time period of events described? • When was this written? • Nationality of author? • What else.. 11-761 22

Heterogeneity • The Dover mail was in its usual genial position that the guard suspected the passengers, the passengers suspected one another and the guard, they all suspected everybody else, and the coachman was sure of nothing but the horses; as to which cattle he could with a clear conscience have taken his oath on the two Testaments that they were not fit for the journey. • News or fiction? • Time period of events described? • When was this written? • Nationality of author? • What else.. 11-761 23

Heterogeneity • The Dover mail was in its usual genial position that the guard suspected the passengers, the passengers suspected one another and the guard, they all suspected everybody else, and the coachman was sure of nothing but the horses; as to which cattle he could with a clear conscience have taken his oath on the two Testaments that they were not fit for the journey. • News or fiction? • Time period of events described? • When was this written? • Nationality of author? • What else.. 11-761 24

Heterogeneity • The Dover mail was in its usual genial position that the guard suspected the passengers, the passengers suspected one another and the guard, they all suspected everybody else, and the coachman was sure of nothing but the horses; as to which cattle he could with a clear conscience have taken his oath on the two Testaments that they were not fit for the journey. • News or fiction? • Time period of events described? • When was this written? • Nationality of author? • What else.. 11-761 25

Who said this? • “Sad” 11-761 26

Heterogeneity in language • What are the various sources of heterogeneity in language 11-761 27

Heterogeneity in language • What are the various sources of heterogeneity in language • How do they differ? 11-761 28

Heterogeneity in language • What are the various sources of heterogeneity in language • How do they differ? • Homework coming up on this problem 11-761 29

True or false • The Merriam Webster dictionary has 470000 words • The Merriam Webster dictionary has over 50 million words 11-761 30

True or false • The Merriam Webster dictionary has 470000 words • The Merriam Webster dictionary has over 50 million words 11-761 31

Types vs. Tokens • Type: Uniquely identifiable value • Words in a lexicon (the left column of the dictionary) • Notes in music (how many) • Token: Instances of types • “Number of words in this article” • “12 notes to a bar” 11-761 32

Word types vs. word tokens • How much wood would woodchuck chuck if woodchuck could chuck wood? As much wood as woodchuck could chuck woodchuck would chuck wood. • Friends, Romans, countrymen, lend me your ears; I come to bury Caesar, not to praise him. The evil that men do lives after them; The good is oft interred with their bones 11-761 33

Frequency of encountering new types • How frequently do we encounter new type as we read the following texts: How much wood would woodchuck chuck if woodchuck could chuck wood? As much wood as woodchuck could chuck woodchuck would chuck wood. Friends, Romans, countrymen, lend me your ears; I come to bury Caesar, not to praise him. The evil that men do lives after them; The good is oft interred with their bones 11-761 34

Frequency of encountering new types • How frequently do we encounter new type as we read the following texts: How much wood would woodchuck chuck if woodchuck could chuck wood? As much wood as woodchuck could chuck woodchuck would chuck wood. Friends, Romans, countrymen, lend me your ears; I come to bury Caesar, not to praise him. The evil that men do lives after them; The good is oft interred with their bones 11-761 35

Frequency of encountering new types • How frequently do we encounter new type as we read the following texts: How much wood would woodchuck chuck if woodchuck could chuck wood? As much wood as woodchuck could chuck woodchuck would chuck wood. Friends, Romans, countrymen, lend me your ears; I come to bury Caesar, not to praise him. The evil that men do lives after them; The good is oft interred with their bones 11-761 36

Frequency of encountering new types • How frequently do we encounter new type as we read the following texts: How much wood would woodchuck chuck if woodchuck could chuck wood? As much wood as woodchuck could chuck woodchuck would chuck wood. Friends, Romans, countrymen, lend me your ears; I come to bury Caesar, not to praise him. The evil that men do lives after them; The good is oft interred with their bones 11-761 37

Frequency of encountering new types • How frequently do we encounter new type as we read the following texts: How much wood would woodchuck chuck if woodchuck could chuck wood? As much wood as woodchuck could chuck woodchuck would chuck wood. Friends, Romans, countrymen, lend me your ears; I come to bury Caesar, not to praise him. The evil that men do lives after them; The good is oft interred with their bones 11-761 38

Frequency of encountering new types • How frequently do we encounter new type as we read the following texts: How much wood would woodchuck chuck if woodchuck could chuck wood? As much wood as woodchuck could chuck woodchuck would chuck wood. Friends, Romans, countrymen, lend me your ears; I come to bury Caesar, not to praise him. The evil that men do lives after them; The good is oft interred with their bones 11-761 39

Frequency of encountering new types • How frequently do we encounter new type as we read the following texts: How much wood would woodchuck chuck if woodchuck could chuck wood? As much wood as woodchuck could chuck woodchuck would chuck wood. Friends, Romans, countrymen, lend me your ears; I come to bury Caesar, not to praise him. The evil that men do lives after them; The good is oft interred with their bones 11-761 40

Frequency of encountering new types • How frequently do we encounter new type as we read the following texts: How much wood would woodchuck chuck if woodchuck could chuck wood? As much wood as woodchuck could chuck woodchuck would chuck wood. Friends, Romans, countrymen, lend me your ears; I come to bury Caesar, not to praise him. The evil that men do lives after them; The good is oft interred with their bones 11-761 41

Frequency of encountering new types • How frequently do we encounter new type as we read the following texts: How much wood would woodchuck chuck if woodchuck could chuck wood? As much wood as woodchuck could chuck woodchuck would chuck wood. Friends, Romans, countrymen, lend me your ears; I come to bury Caesar, not to praise him. The evil that men do lives after them; The good is oft interred with their bones 11-761 42

Frequency of encountering new types • How frequently do we encounter new type as we read the following texts: How much wood would woodchuck chuck if woodchuck could chuck wood? As much wood as woodchuck could chuck woodchuck would chuck wood. Friends, Romans, countrymen, lend me your ears; I come to bury Caesar, not to praise him. The evil that men do lives after them; The good is oft interred with their bones 11-761 43

Frequency of encountering new types • How frequently do we encounter new type as we read the following texts: How much wood would woodchuck chuck if woodchuck could chuck wood? As much wood as woodchuck could chuck woodchuck would chuck wood. Friends, Romans, countrymen, lend me your ears; I come to bury Caesar, not to praise him. The evil that men do lives after them; The good is oft interred with their bones 11-761 44

Frequency of encountering new types • How frequently do we encounter new type as we read the following texts: How much wood would woodchuck chuck if woodchuck could chuck wood? As much wood as woodchuck could chuck woodchuck would chuck wood. Friends, Romans, countrymen, lend me your ears; I come to bury Caesar, not to praise him. The evil that men do lives after them; The good is oft interred with their bones 11-761 45

Frequency of encountering new types • How frequently do we encounter new type as we read the following texts: How much wood would woodchuck chuck if woodchuck could chuck wood? As much wood as woodchuck could chuck woodchuck would chuck wood. Friends, Romans, countrymen, lend me your ears; I come to bury Caesar, not to praise him. The evil that men do lives after them; The good is oft interred with their bones 11-761 46

Frequency of encountering new types • How frequently do we encounter new type as we read the following texts: How much wood would woodchuck chuck if woodchuck could chuck wood? As much wood as woodchuck could chuck woodchuck would chuck wood. Friends, Romans, countrymen, lend me your ears; I come to bury Caesar, not to praise him. The evil that men do lives after them; The good is oft interred with their bones 11-761 47

Frequency of encountering new types • How frequently do we encounter new type as we read the following texts: How much wood would woodchuck chuck if woodchuck could chuck wood? As much wood as woodchuck could chuck woodchuck would chuck wood. Friends, Romans, countrymen, lend me your ears; I come to bury Caesar, not to praise him. The evil that men do lives after them; The good is oft interred with their bones 11-761 48

Frequency of encountering new types • How frequently do we encounter new type as we read the following texts: How much wood would woodchuck chuck if woodchuck could chuck wood? As much wood as woodchuck could chuck woodchuck would chuck wood. Friends, Romans, countrymen, lend me your ears; I come to bury Caesar, not to praise him. The evil that men do lives after them; The good is oft interred with their bones 11-761 49

Frequency of encountering new types • How frequently do we encounter new type as we read the following texts: How much wood would woodchuck chuck if woodchuck could chuck wood? As much wood as woodchuck could chuck woodchuck would chuck wood. Friends, Romans, countrymen, lend me your ears; I come to bury Caesar, not to praise him. The evil that men do lives after them; The good is oft interred with their bones 11-761 50

Frequency of encountering new types • How frequently do we encounter new type as we read the following texts: How much wood would woodchuck chuck if woodchuck could chuck wood? As much wood as woodchuck could chuck woodchuck would chuck wood. Friends, Romans, countrymen, lend me your ears; I come to bury Caesar, not to praise him. The evil that men do lives after them; The good is oft interred with their bones 11-761 51

Frequency of encountering new types • How frequently do we encounter new type as we read the following texts: How much wood would woodchuck chuck if woodchuck could chuck wood? As much wood as woodchuck could chuck woodchuck would chuck wood. Friends, Romans, countrymen, lend me your ears; I come to bury Caesar, not to praise him. The evil that men do lives after them; The good is oft interred with their bones 11-761 52

Frequency of encountering new types • How frequently do we encounter new type as we read the following texts: How much wood would woodchuck chuck if woodchuck could chuck wood? As much wood as woodchuck could chuck woodchuck would chuck wood. Friends, Romans, countrymen, lend me your ears; I come to bury Caesar, not to praise him. The evil that men do lives after them; The good is oft interred with their bones 11-761 53

Frequency of encountering new types • How frequently do we encounter new type as we read the following texts: How much wood would woodchuck chuck if woodchuck could chuck wood? As much wood as woodchuck could chuck woodchuck would chuck wood. Friends, Romans, countrymen, lend me your ears; I come to bury Caesar, not to praise him. The evil that men do lives after them; The good is oft interred with their bones 11-761 54

Frequency of encountering new types • How frequently do we encounter new type as we read the following texts: How much wood would woodchuck chuck if woodchuck could chuck wood? As much wood as woodchuck could chuck woodchuck would chuck wood. Friends, Romans, countrymen, lend me your ears; I come to bury Caesar, not to praise him. The evil that men do lives after them; The good is oft interred with their bones 11-761 55

Frequency of encountering new types • How frequently do we encounter new type as we read the following texts: How much wood would woodchuck chuck if woodchuck could chuck wood? As much wood as woodchuck could chuck woodchuck would chuck wood. Friends, Romans, countrymen, lend me your ears; I come to bury Caesar, not to praise him. The evil that men do lives after them; The good is oft interred with their bones 11-761 56

Type-token curves ntypes ntokens • Typical type-token curve • Increases monotonically • Gets flatter all the time • But never gets completely flat • There are always new words you will encounter • Type-token curves will differ for different sub languages 11-761 57

Comparing type-token curves for sublanguages • “Wall Street Journal” Corpus (WSJ): • Newspaper articles, 1988-1992 • Written English, rich vocabulary (leaning towards finance) • “Switchboard” Corpus (SWB): • Transcribed spoken conversations • over the telephone • Prescribed topic (one of 70) • 1990’s • “Broadcast News” Corpus (BN): • Transcribed TV/Radio News programs • Spoken, but somewhat scripted

Comparing type-token curves for sub-languages 11-761 59

WSJ vs BN vs SWB (log scale) Note: slope << 1 60

Token-type curves: Bigrams • The number of bigrams is greater than unigrams • The probability of hitting a “new” bigram type is higher • The curve is steeper, but flattens out after a few tens of millions of tokens • Distinctions between sub-languages is more stark 61

Bigram Token Type Curve – BN vs. SWB (log scale) Note: slope closer to 1.0 than for unigrams

Token-type curves: Trigrams • The number of trigrams is greater than bigrams • The curve is steeper than for bigrams Will flatten out after hundreds of millions of tokens • 11-761 63

Trigram Token-Type Curve – BN vs. SWB (log scale) Note: slope almost 1.0

Head of word-frequency lists Count unit: 1000 • WSJ vs BN vs Switchboard 11-761 65

Tail of word-frequency lists Singletons (Count = 1) • WSJ vs BN vs Switchboard 11-761 66

Sub-language Example 2 • The Diabetes set includes 9 Diabetes-related journals and a total of 4.5M tokens and 95K types. • The Veterinary science set includes 11 journals and 3.2M tokens and 87K types. • All Journals were extracted from PubMed in Oct 2010 and they include everything that was available by those journals up until then. • This example is provided by Dana Movshovitz-Attias.

Diabetes vs. Veterinary: Type-Token Curve

Diabetes vs. Veterinary: Type-Token Curve (log scale)

Head of Word Frequency List (counts per 1,000 tokens) diabetes count veterinary count Count unit: 1000 THE 42 THE 57 OF 35 OF 39 AND 31 AND 30 IN 29 IN 29 TO 16 TO 17 WITH 13 A 14 A 13 WERE 11 FOR 10 WAS 10 WAS 10 FOR 10 WERE 9 WITH 9 DIABETES 7 FROM 7 THAT 7 THAT 6 BY 6 IS 6 IS 6 AS 6 2 6 BY 6 AS 5 ON 5 INSULIN 5 AT 5 OR 5 1 4 GLUCOSE 5 BE 4 1 5 THIS 4

Tail of Word Frequency List: Count=1 (“Singletons”) Diabetes Veterinary QUESTIONNAIRE-BASED MOLARITIES CAPACITY-CONSTRAINED LIDOCAIN Singletons DND MULTIORGAN (Count = 1) 1003500 MICROGLIA-MEDIATED ENZYME-INHIBITOR NALYSIS ALVEOLUS-CAPILLARY 10702 KUZUYA BLUE-DNA $6054 HAIR-LOSS SENTENCING POPULATION-DYNAMICAL PAPER-AND-PENCIL STATE-TRANSITION

An interesting feature • In every case, word frequency count goes down very fast • Lets plot the relative frequency of words against their rank.. 11-761 72

P(word) vs rank • The probability of a word falls off rapidly with rank! • This is an instance of a more generic principle.. 11-761 73

A peculiar phenomenon.. • There are many more rare things than there are common things! • This is true, not just of words.. 11-761 74

A peculiar phenomenon.. • There are many more rare things than there are common things! • This is true, not just of words.. • In any sufficiently large collection.. • The most frequent event is • ~2 times as frequent as the second most frequent event • ~3 times as frequent as the fourth most frequent event • ~4 times as frequent as the fourth most frequent event • .. • ~N times as frequent as the N-th most frequent event 11-761 75

Typical behavior ! Frequency !/2 !/3 !/4 !/5 !/6 !/7 !/8 !/9 !/10 1 2 3 4 5 6 7 8 9 10 Rank • Rank vs relative frequency 11-761 76

Typical behavior in Log domain • Rank vs relative frequency • In a log-log plot its just a line with negative slope approximately 1 11-761 77

Examples: Population of cities • Caveat: Axes are flipped w.r.t. earlier figure • The most populous city is approx. twice as populous as the second most 78 populous city and so on..

Examples: AOL users vs. sites • AOL visitors to sites • http://www.hpl.hp.com/research/idl/papers/ranking/ranking.html 79

Examples: Cryptocurrencies • https://steemit.com/steem/@akrid/applying-zipf-s-law- to-the-crypto-market 80

Examples: UNK • I think this is an example • No clue what it is.. 81

And, of course, words.. • Word counts in wikipedias of 30 languages • (from Wikipedia) 11-761 82

Zipf’s law George Kingsley Zipf (1902-1950) Linguist • Define the probability of a word in terms of its rank • Zipf’s hypothesis 1 ! "#$% ∝ $()*("#$%) • This is an empirical law 11-761 83

Zipf’s law log !("#$%) !("#$%) 1 3 5 7 9 11 13 log $()*("#$%) $()*("#$%) 1 ! "#$% ∝ $()*("#$%) • In a log-log plot the relationship is linear log !("#$%) = 1 − log($()*("#$%)) • The slope of the plot is -1.0. 11-761 84

Frequency vs. rank, Brown Corpus • Brown Corpus (1969) 500 samples of English-language text, totaling roughly one million words, compiled from • works published in the United States in 1961 15 text categories • • Appears to match Zipf 85

Is Zipf’s distribution valid? !("#$%) 1 3 5 7 9 11 13 $()*("#$%) • Is the following a valid distribution? 1 ! "#$% ∝ $()*("#$%) • Correction: 1 ! "#$% ∝ $()*("#$%) -./ 11-761 86

Adjustment !("#$%) 1 3 5 7 9 11 13 $()*("#$%) • Is the following a valid distribution? 1 ! "#$% ∝ $()*("#$%) • Only for finite vocabularies • Correction for infinite vocabularies: 1 ! "#$% ∝ $()*("#$%) -./ 87

Word unigrams, WSJ corpus • Word frequency vs rank, WSJ corpus 11-761 88

Word bigrams, WSJ corpus • Word bigram frequency vs rank, WSJ corpus 11-761 89

Word trigrams, WSJ corpus • Word trigram frequency vs rank, WSJ corpus 11-761 90

Word trigrams, WSJ corpus • Falls off rapidly with rank as predicted 11-761 91

Zipf’s law • Zipf’s law is seen to apply to a large variety of natural phenomena • Zipf suggested it is the outcome of the “principle of least effort” • We – and nature – spends the least effort most of the time • Short, easy (or easy-to-recall) words are used most frequently. Longer harder words are less frequent. • But a more mathematical explanation came from Beniot Mandelbrot and George Miller 11-761 92

Zipf’s law: Monkey on a typewriter • A monkey on a keyboard will produce text that follows Zipf’s law • Shorter words are more likely than longer words. If the keyboard has only 26 characters + space: ! "#$%&ℎ = ) = 26 , 27 ,./ 1 ! 0123 | "#$%&ℎ = ) = 26 , • Combining these will give you Zipf’s law (not quite; but we return to this) • Problem: Language is not a monkey on a typewriter 93

Don’t try this at home From wikipedia In 2003, lecturers and students from the University of Plymouth MediaLab Arts course used a £2,000 grant from the Arts Council to study the literary output of real monkeys. They left a computer keyboard in the enclosure of six Celebes crested macaques in Paignton Zoo in Devon in England for a month, with a radio link to broadcast the results on a website… Not only did the monkeys produce nothing but five total pages largely consisting of the letter S, the lead male began by bashing the keyboard with a stone, and the monkeys continued by urinating and defecating on it. 94

The Pareto distribution Pareto type 1 & ! " = $" % " &'( )*+ " ≥ " % Discovered by Vilfredo Pareto 1848-1923 • Real life phenomena are concentrated into a few frequent types, with a long tail of infrequent ones. 20% of the values account for 80% of the probability mass. • Almost everything in the universe follows the Pareto distribution • The distribution of wealth (most money in the hands of a few) • The sizes of human settlements (few cities, many hamlets/villages) • File size distribution of Internet traffic which uses the TCP protocol (many smaller files, few larger ones) • Hard disk drive error rates • The values of oil reserves in oil fields (a few large fields, many small fields) 11-761 95 • The sizes of sand particles

Zipf’s law: The Pareto distribution • Matthew 13 : 12 For whosoever hath, to him shall be given, and he shall have more abundance: but whosoever hath not, from him shall be taken away even that he hath. • When you use a word once, its easier to recall, so you are more likely to use it again • Over time, you will use familiar words more and more frequently and unfamiliar ones less and less • The same principle applies to other data 11-761 96

Mandelbrot.. • Beniot Mandelbrot (1924-2010) • Fractal theory, Chaos theory, Mandelbrot-zipf law 11-761 97

Mandelbrot-Zipf law • The money on the typewriter doesn’t actually produce Zipf’s distribution • Instead you get the Mandelbrot distribution $ ! " = % + '()*(") - • Zipf’s rule is a special case if % = 0 and / = 1 11-761 98

Mandelbrot distribution $ ! " = % + '()*(") - • In logs log ! " = ( − 2 log % + '()*(") • No longer exactly a line (curves “down”) • 2 changes the slope of the curve • % changes the log transform itself • For low-rank " , it allows you to start from a relatively high log transform • Better fits most data 11-761 99

These curves (Unigram).. • Actually better fit the Mandelbrot distribution for different values of ! and " 11-761 100