SLIDE 1
The pope is catholic.
language as an interface language as data
introduction
Language as an Interface Spencer Kelly introduction The pope is - - PowerPoint PPT Presentation
Language as an Interface Spencer Kelly introduction The pope is - - PowerPoint PPT Presentation
Language as an Interface Spencer Kelly introduction The pope is catholic. language as data language as an interface introduction (@spencermountain) introduction problem problem problem problem 4-gram: 3-gram: 2-gram: 1-gram: london
SLIDE 2
SLIDE 3
(@spencermountain)
introduction
SLIDE 4
introduction
SLIDE 5
problem
SLIDE 6
problem
SLIDE 7
problem
SLIDE 8
london in the rain london in the in the rain london in in the the rain london in the rain
4-gram: 3-gram: 2-gram: 1-gram:
10 requests per keystroke
4-words -
5-words : 15 6-words : 21 7-words : 28 8-words : 36
problem
SLIDE 9
london in the rain london in the in the rain london in in the the rain London in the rain in the london in the in the rain london in the rain
Stopwords blacklist: Edge gram filter: Redundancy check:
#1 #2 #3
in the
“rain” “london”
“london in the rain”
problem
SLIDE 10
- NLTK - excellent, huge, python
- Stanford parser - excellent, huge, java
Alchemy, TextRazor, OpenCalais, Embedly, Zemanta Or an offsite API?
- Freeling - excellent, huge, C++
- Illinois tagger - excellent, huge, java
When all you’ve got is a jackhammer..
niche
SLIDE 11
Can it be hacked? tldr: yes.
niche
SLIDE 12
Zipfs law
The top 10 words account for 25% of language. The top 100 words account for 50% of language. The top 50,000 words account for 95% of language.
niche
SLIDE 13
How big is a language? Shakespeare - 35,000 Wordnet - 200,000 ! OED - 600,000 !
niche
SLIDE 14
602 kb
uncompressed
50,000 different words
An average person will ever hear ~4 lookups in binary search
niche
SLIDE 15
first, let’s kill the nouns 70%
process
180 kb
uncompressed
SLIDE 16
Noun Verb Adjective Adverb Tomato Tomatoes Toronto Torontonian Speak Spoke Speaking will speak have spoken had spoken ... nice nicer nicest quickly quicklier quickliest “awesome” “awesomeify” improveify your vocabularies “quickly” “quick”
n/2.3
Each word
*not handsome *not truly *not is *not economics
“tomatoey” “tomato” “speak” “speaker” “aggressive” “agressiveness” “civil” “civilize”
niche
SLIDE 17
then, let’s conjugate
- ur verbs
process
110 kb
uncompressed
SLIDE 18
process
jQuery
256kb
d3js
330kb
react
653kb
110 kb
uncompressed
lodash
503kb
the whole english language
110kb
SLIDE 19
Ok, let’s roll our own POS tagger..
(what could go rong?)
process
SLIDE 20
1) Lexicon 2) Suffix regexes 3) Sentence-level markov chain
SLIDE 21
Suffix rules
process
SLIDE 22
Grammar rules - markov
She could walk the walk .
before: Verb - Det - Verb after: Verb - Det - Noun
process
SLIDE 23
“Unreasonable effectiveness” of rule-based taggers-
- a 1,000 word lexicon - 45% precision
- fallback to [Noun] - 70% precision
- a little regex - 74% precision
- a little grammar in it - 81% precision
process
SLIDE 24
t.text(“keep on rocking in the free world”) t.negate() //“don’t keep on rocking in the free world.”
- utcome
SLIDE 25
t.text(“it is a cool library”) t.toValleyGirl() //“so, it is like, a cool library.”
- utcome
SLIDE 26
We gave the monkeys the bananas,
..because they were ripe. ..because they were hungry.
- utcome
SLIDE 27
We gave the monkeys the bananas [Pr] [Verb] [Dt] [Noun] [Dt] [Noun]
list of letters POS-tagging Dependency parser
We give [Noun] [Noun]
[act / transfer / voluntary] [genus / monkey]
Knowledge engine
[plant / banana]
- utcome
SLIDE 28
#TODOFML
- Mutable/Immutable API
- Speed, performance testing
- Romantic-language verb conjugations
- ‘bl.ocks.org’ of demos and docs
- utcome
SLIDE 29
npm install --wooyeah @spencermountain
Slack group, mailing list, github, Toronto/coffee
SLIDE 30