Information Theory And Language Romain Brasselet, SISSA 09/07/15 - - PowerPoint PPT Presentation

▶
information theory and language
SMART_READER_LITE
LIVE PREVIEW

Information Theory And Language Romain Brasselet, SISSA 09/07/15 - - PowerPoint PPT Presentation

Mar 22, 2023 167 likes •832 views

Information Theory And Language Romain Brasselet, SISSA 09/07/15 Framework of Information Theory Exemplified Exemplified Entropy and redundancy of written language Space of letters r h n w k e c j q y a x p l z m g v t

slide-1
SLIDE 1

Information Theory And Language

Romain Brasselet, SISSA

09/07/15

slide-2
SLIDE 2
slide-3
SLIDE 3

Framework of Information Theory

slide-4
SLIDE 4

Exemplified

slide-5
SLIDE 5

Exemplified

slide-6
SLIDE 6

Entropy and redundancy

  • f written language
slide-7
SLIDE 7

a b c d e f g h i j k l m n

  • p

q r s t u v w x y z

→ no correlation analysis

Space of letters

slide-8
SLIDE 8

Probabilities

a

b

c

d

e

f

g

h

j

k

l

m

n

  • p
q

r

s

t

u

v

w

x

y

z

i

slide-9
SLIDE 9

Entropy as a measure of uncertainty and information

slide-10
SLIDE 10

Entropy of a coin

slide-11
SLIDE 11

Entropy of language?

slide-12
SLIDE 12

Redundancy

Redundancy ~ structure

Language has structure and therefore is redundant.

slide-13
SLIDE 13

Conditional probabilities

if you really want to hear about it the first thing youll probably want to know is where i was born

slide-14
SLIDE 14

Catcher in the Rye

if you really want to hear about it the first thing you ll probably want to know is where i was born an what my lousy childhood was like and how my parents were occupied and all before they had me and all that david copperfield kind of crap but i don t feel like going into it if you want to know the truth in the first place that stuff bores me and in the second place my parents would have about two hemorrhages apiece if i told anything pretty personal about them they re quite touchy about anything like that especially my father they re nice and all i m not saying that but they re also touchy as hell besides i m not going to tell you my whole goddam autobiography or anything i ll just tell you about this madman stuff that happened to me around last christmas just before i got pretty run down and had to come out here and take it easy i mean that s all i told db about and he s my brother and all he s in hollywood that isn t too far from this crumby place and he comes over and visits me...

slide-15
SLIDE 15

Joint probabilities

slide-16
SLIDE 16

Joint probabilities

slide-17
SLIDE 17

Joint probabilities

slide-18
SLIDE 18

Conditional probabilities

slide-19
SLIDE 19

Conditional entropy

slide-20
SLIDE 20

Conditional probabilities

slide-21
SLIDE 21

Conditional entropy

slide-22
SLIDE 22

Entropies

slide-23
SLIDE 23

Generation of sentences

myig ohi lunnh p mtoswers h oc llwdn cdsieal tihd r hhhicggnd w daeasereeoynth iar iehttiomlmele dazoo toede orhsiuee adfatc tfku u uahtd lk tninnorn ena tod oof tualm lletnsth qiiwoetli s esd t 2 4

slide-24
SLIDE 24

Generation of sentences

the chat agodding ancid nier ove m fen hin aftelee diall or ando an s jusea pen he not

  • nting whame the new a sup everse mides he it inee s have ve way i wit she my wit

kictle th cradlay to fave sorriven thembeets bally heintice goddamearobvin onsted i loozencey got hating bon the ater hell the bouldiew hat king ught mid her a pread ing yout did hand he teeng like hels and peng abou myig ohi lunnh p mtoswers h oc llwdn cdsieal tihd r hhhicggnd w daeasereeoynth iar iehttiomlmele dazoo toede orhsiuee adfatc tfku u uahtd lk tninnorn ena tod oof tualm lletnsth qiiwoetli s esd t 2 4

slide-25
SLIDE 25

Generation of sentences

the crumby bar when i got him except giving out gear her and running teachests at pretty were this guts i could hartzell over man keep you re you happened about a handshaking her i have one of stuff they probably hurt sort of my hardy up at the was the d even he hardly guy right and parents were s goddam hound none comed and that we got booth the chat agodding ancid nier ove m fen hin aftelee diall or ando an s jusea pen he not

  • nting whame the new a sup everse mides he it inee s have ve way i wit she my wit

kictle th cradlay to fave sorriven thembeets bally heintice goddamearobvin onsted i loozencey got hating bon the ater hell the bouldiew hat king ught mid her a pread ing yout did hand he teeng like hels and peng abou myig ohi lunnh p mtoswers h oc llwdn cdsieal tihd r hhhicggnd w daeasereeoynth iar iehttiomlmele dazoo toede orhsiuee adfatc tfku u uahtd lk tninnorn ena tod oof tualm lletnsth qiiwoetli s esd t 2 4

slide-26
SLIDE 26

Entropies

How do we go further?

slide-27
SLIDE 27
slide-28
SLIDE 28

Shannon's guessing game 1

slide-29
SLIDE 29

Shannon's guessing game 2

slide-30
SLIDE 30

Shannon's guessing game 2

what letter? → what guess?

slide-31
SLIDE 31

Guessing game

slide-32
SLIDE 32

Entropy of written english

slide-33
SLIDE 33

However...

The entropy of the code depends on the writer. The guessing game depends on the knowledge

  • f the reader.

Cover and King, IEEE transactions on Information Theory 1978

slide-34
SLIDE 34

Source coding theorem

slide-35
SLIDE 35

Importance of redundancy

  • Redundancy is a measure of how efficiently symbols are

used.

  • It is a sign of structure in the language.
  • It reduces communication rate but increases predictability.
  • Redundancy allows us to reconstruct noisy signals:

“Turn phat mufic down”

  • We can see language as a compromise between

information and redundancy.

slide-36
SLIDE 36

Zipf's law

slide-37
SLIDE 37

Word entropy

slide-38
SLIDE 38
slide-39
SLIDE 39

Entropy of word ordering

slide-40
SLIDE 40

Entropy of word ordering

slide-41
SLIDE 41

Model of word formation

slide-42
SLIDE 42
slide-43
SLIDE 43

Model

Consider a population of individuals who can communicate via signals. Signals may include gestures, facial expressions, or spoken sounds. Each individual is described by an active matrix P and a passive matrix Q. The entry P denotes that the probability that the individual, as a speaker, will refer to object i by using signal j. The entry Q denotes the probability that the individual, as a listener, will interpret signal j as referring to object i.

slide-44
SLIDE 44

Model

P Q'

slide-45
SLIDE 45

Model

P Q U

slide-46
SLIDE 46
slide-47
SLIDE 47

Noise/confusion

  • Languages whose basic signals consist of m phonemes.
  • The words of the language are all l-phonemes long.
  • The probability of confusion between words is defined by the

product of the probability of confusion of their phonemes.

Where are the phonemes.

slide-48
SLIDE 48

P emitter matrix

slide-49
SLIDE 49

Miller & Nicely 1955

slide-50
SLIDE 50

Miller & Nicely 1955

slide-51
SLIDE 51

U noise matrix

slide-52
SLIDE 52

What is the optimal Q passive matrix?

First, a guess: a listener should interpret perceived output word w as

  • bject i with a probability which equals the probability

that, when trying to communicate object i, the perceived

  • utput would be w.
slide-53
SLIDE 53

Q receiver matrix

slide-54
SLIDE 54

Fitness as a function of noise

slide-55
SLIDE 55

Maximum likelihood Q matrix

slide-56
SLIDE 56

Fitness as a function of noise

slide-57
SLIDE 57

Word formation

slide-58
SLIDE 58

Theorem

where

slide-59
SLIDE 59

Theorem

slide-60
SLIDE 60

Word formation

Of course, in reality, words don't grow arbitrarily longer. But they still permit a decrease in the error rate.

slide-61
SLIDE 61

Evolution of syntax

slide-62
SLIDE 62
slide-63
SLIDE 63

Word learning

where = abundance of word in population

slide-64
SLIDE 64

To syntax or not to syntax?

slide-65
SLIDE 65

To syntax or not to syntax?

O+A O+A O+A W W W

slide-66
SLIDE 66