SLIDE 1
Weakly-Supervised Bayesian Learning of a CCG Supertagger
Dan Garrette, Chris Dyer, Jason Baldridge, Noah A. Smith
Weakly-Supervised Bayesian Learning of a CCG Supertagger Dan - - PowerPoint PPT Presentation
Weakly-Supervised Bayesian Learning of a CCG Supertagger Dan - - PowerPoint PPT Presentation
Weakly-Supervised Bayesian Learning of a CCG Supertagger Dan Garrette, Chris Dyer, Jason Baldridge, Noah A. Smith Type-Level Supervision Type-Level Supervision Unannotated text Incomplete tag dictionary: word {tags} Type-Level
SLIDE 2
SLIDE 3
Type-Level Supervision
- Unannotated text
- Incomplete tag dictionary: word ↦ {tags}
SLIDE 4
Type-Level Supervision
Used for POS tagging for 20+ years
[Kupiec, 1992] [Merialdo, 1994]
SLIDE 5
Type-Level Supervision
Good POS tagger performance even with low supervision
[Das & Petrov 2011] [Garrette & Baldridge 2013] [Garrette et al. 2013]
SLIDE 6
Combinatory Categorial Grammar (CCG)
SLIDE 7
CCG
Every word token is associated with a category Categories combine to categories of constituents
[Steedman, 2000] [Steedman and Baldridge, 2011]
SLIDE 8
the dog n np n / np np
CCG
SLIDE 9
s dogs np sleep \ s s
CCG
np
SLIDE 10
POS vs. Supertags
the dog sleeps np/n n s\np np s the dog sleeps DT NN VBZ NP S VP
SLIDE 11
Supertagging
Type-supervised learning for supertagging is much more difficult than for POS Penn Treebank POS CCGBank Supertags 48 tags 1,239 tags
SLIDE 12
CCG
The grammar formalism itself can be used to guide learning
SLIDE 13
CCG Supertagging
SLIDE 14
- Sequence tagging problem, like POS-tagging
- Building block for grammatical parsing
CCG Supertagging
SLIDE 15
Supertagging
“almost parsing”
[Bangalore and Joshi 1999]
SLIDE 16
sleeps the lazy dog
Why Supertagging?
/ np n n / n n \ s np
SLIDE 17
n np n np np n sleeps the lazy dog np / n n / \ s s s
Why Supertagging?
SLIDE 18
n np np np n sleeps the lazy dog / n n / \ s np s n n
CCG Supertagging
SLIDE 19
n np np np n sleeps the lazy dog / n n / \ s np s n
CCG Supertagging
SLIDE 20
sleeps the lazy dog s\np n
CCG Supertagging
n / n np n /
SLIDE 21
the lazy dog np/n ? n
CCG Supertagging
SLIDE 22
Principle #1
the lazy dog np/n n np X X
Prefer Connections
SLIDE 23
Supertags vs. POS
the dog sleeps np/n n s\np np s the dog sleeps DT NN VBZ NP S VP universal, intrinsic grammar properties all relationships must be learned ?
SLIDE 24
the lazy dog np/n (np\(np/n))/n n
Principle #2 Prefer Simplicity
SLIDE 25
Prefer Simplicity
buy := (((sb\np)/pp)/pp)/np appears 342 times in CCGbank buy := (sb\np)/np appears once
e.g. “Opponents don't buy such arguments.” “Tele-Communications agreed to buy half of Showtime Networks from Viacom for $ 225 million.” pp pp
SLIDE 26
a {s, np, n,…} A B / B A B \ B patom(a) A A B \ C B / C × pterm pterm pterm pterm pterm × pfwd × pfwd × pfwd × pfwd × pmod × pmod × pmod × pmod
Weighted Tag Grammar
SLIDE 27
the lazy dog np/n n/n n (np\(np/n))/n np
CCG Supertagging
SLIDE 28
HMM Transition Prior
P(t → u) = λ · P(u) + (1−λ) · P(t → u)
simple is good connecting is good
SLIDE 29
Type-Supervised Learning
unlabeled corpus tag dictionary universal properties of the CCG formalism
same as POS tagging
SLIDE 30
Training
SLIDE 31
Posterior Inference
Forward-Filter Backward-Sample (FFBS)
- [Carter and Kohn, 1996]
SLIDE 32
Posterior Inference
the lazy dogs wander np/n n np n/n np (s\np)/np n n/n np/n s\np …
Tag Dictionary ___ : __, __, __ ___ : __, __, __ ___ : __, __, __ ___ : __, __, __ ___ : __, __, __ Unlabeled Data ______________ ______________ ______________ ______________ ______________
SLIDE 33
Posterior Inference
the lazy dogs np/n n np n/n np wander (s\np)/np n n/n np/n s\np …
Priors HMM
SLIDE 34
Posterior Inference
the lazy dogs np/n n np n/n np wander (s\np)/np n n/n np/n s\np …
Priors HMM
SLIDE 35
Posterior Inference
the lazy dogs np/n n np n/n np wander (s\np)/np n n/n np/n s\np …
Priors HMM
SLIDE 36
Posterior Inference
the lazy dogs np np wander (s\np)/np n n/n np/n …
Priors HMM
np/n n n/n s\np np/n n n/n s\np
SLIDE 37
Posterior Inference
the lazy dogs np np wander (s\np)/np n n/n np/n …
Priors HMM
np/n n n/n s\np
SLIDE 38
Experiments
SLIDE 39
Baldridge 2008
Use universal properties of CCG to initialize EM
- Simpler definition of category complexity
- No corpus-specific information
SLIDE 40
English Supertagging
25 50 75 100 0.1 0.01 0.001 none
Baldridge '08 Ours
78 80 80 67 55 73 41 51 tag dictionary pruning cutoff t a g g i n g a c c u r a c y
SLIDE 41
Chinese Supertagging
25 50 75 100 0.1 0.01 0.001 none
Baldridge '08 Ours
66 69 62 43 33 56 28 49 tag dictionary pruning cutoff t a g g i n g a c c u r a c y
SLIDE 42
Italian Supertagging
25 50 75 100 0.1 0.01 0.001 none
Baldridge '08 Ours
54 53 47 36 33 45 32 46 tag dictionary pruning cutoff t a g g i n g a c c u r a c y
SLIDE 43
Code Available
GitHub repository linked from my website
SLIDE 44
Conclusion
Combining annotation exploitation with universal grammatical knowledge yields good models from weak supervision