reranking and self training for parser adaptation
play

Reranking and Self-Training for Parser Adaptation David McClosky, - PowerPoint PPT Presentation

Sep 23, 2022 •163 likes •583 views

Reranking and Self-Training for Parser Adaptation David McClosky, Eugene Charniak, and Mark Johnson { dmcc|ec|mj } @cs.brown.edu Brown Laboratory for Linguistic Information Processing (BLLIP) David McClosky - dmcc@cs.brown.edu - ACL 2006 -

  1. Reranking and Self-Training for Parser Adaptation David McClosky, Eugene Charniak, and Mark Johnson { dmcc|ec|mj } @cs.brown.edu Brown Laboratory for Linguistic Information Processing (BLLIP) David McClosky - dmcc@cs.brown.edu - ACL 2006 - 7.18.2006 - 1

  2. Overview Introduction and Previous Work Parser portability Parser adaptation Reranker portability Analysis Future Work and Conclusions David McClosky - dmcc@cs.brown.edu - ACL 2006 - 7.18.2006 - 2

  3. Parsing David McClosky - dmcc@cs.brown.edu - ACL 2006 - 7.18.2006 - 3

  4. Parameters Parser as in [Charniak and Johnson ACL 2005] Corpus # words # sentences Parameters 950,028 39,832 ∼ 2,200,000 WSJ 373,152 19,740 ∼ 1,300,000 BROWN Number of parameters is a function of training data. David McClosky - dmcc@cs.brown.edu - ACL 2006 - 7.18.2006 - 4

  5. Parsing David McClosky - dmcc@cs.brown.edu - ACL 2006 - 7.18.2006 - 5

  6. n -best Parsing David McClosky - dmcc@cs.brown.edu - ACL 2006 - 7.18.2006 - 6

  7. Reranking Parsers David McClosky - dmcc@cs.brown.edu - ACL 2006 - 7.18.2006 - 7

  8. More Parameters Reranking parser as in [Charniak and Johnson 2005] 14 feature schemas Extract features according to schemas then estimate feature weights Corpus Parser parameters Reranker features ∼ 2,200,000 ∼ 1,300,000 WSJ ∼ 1,300,000 ∼ 700,000 BROWN Again, number of parameters is a function of training data. David McClosky - dmcc@cs.brown.edu - ACL 2006 - 7.18.2006 - 8

  9. Corpora and Domains WSJ : labeled news text, about 40,000 parses NANC : unlabeled news text, about 24 million sentences BROWN : labeled text from various domains, about 24,000 parses total David McClosky - dmcc@cs.brown.edu - ACL 2006 - 7.18.2006 - 9

  10. Corpora and Domains WSJ : labeled news text, about 40,000 parses NANC : unlabeled news text, about 24 million sentences BROWN : labeled text from various domains, about 24,000 parses total Divisions as in [Bacchiani et al. 2006] (based on [Gildea 2001]) 19,740 train, 2,078 tune, 2,425 test Treebanked sections are predominantly fiction Each division of the corpus consists of sentences from all available genres David McClosky - dmcc@cs.brown.edu - ACL 2006 - 7.18.2006 - 9

  11. Self-Training [McClosky, Charniak, and Johnson NAACL 2006] Train model from labeled data train reranking parser on WSJ Use model to annotate unlabeled data use model to parse NANC Combine annotated data with labeled training data merge parsed NANC data with WSJ training data Train a new model from the combined data train reranking parser on WSJ + NANC data David McClosky - dmcc@cs.brown.edu - ACL 2006 - 7.18.2006 - 10

  12. Overtrained? Question: How does setting so many parameters from Wall Street Journal data affect parsing performance on the Brown corpus? David McClosky - dmcc@cs.brown.edu - ACL 2006 - 7.18.2006 - 11

  13. Previous Work f -measure Training Testing Gildea Bacchiani 86.4 87.0 WSJ WSJ 80.6 81.1 WSJ BROWN 84.0 84.7 BROWN BROWN WSJ + BROWN 84.3 85.6 BROWN [Gildea 2001] , [Bacchiani et al. 2006] David McClosky - dmcc@cs.brown.edu - ACL 2006 - 7.18.2006 - 12

  14. Previous Work f -measure Training Testing Gildea Bacchiani 86.4 87.0 WSJ WSJ 80.6 81.1 WSJ BROWN 84.0 84.7 BROWN BROWN WSJ + BROWN 84.3 85.6 BROWN [Gildea 2001] , [Bacchiani et al. 2006] David McClosky - dmcc@cs.brown.edu - ACL 2006 - 7.18.2006 - 12

  15. Previous Work f -measure Training Testing Gildea Bacchiani 86.4 87.0 WSJ WSJ 80.6 81.1 WSJ BROWN 84.0 84.7 BROWN BROWN WSJ + BROWN 84.3 85.6 BROWN [Gildea 2001] , [Bacchiani et al. 2006] David McClosky - dmcc@cs.brown.edu - ACL 2006 - 7.18.2006 - 12

  16. Previous Work f -measure Training Testing Gildea Bacchiani 86.4 87.0 WSJ WSJ 80.6 81.1 WSJ BROWN 84.0 84.7 BROWN BROWN WSJ + BROWN 84.3 85.6 BROWN [Gildea 2001] , [Bacchiani et al. 2006] David McClosky - dmcc@cs.brown.edu - ACL 2006 - 7.18.2006 - 12

  17. Previous Work f -measure Training Testing Gildea Bacchiani 86.4 87.0 WSJ WSJ 80.6 81.1 WSJ BROWN 84.0 84.7 BROWN BROWN WSJ + BROWN 84.3 85.6 BROWN [Gildea 2001] , [Bacchiani et al. 2006] David McClosky - dmcc@cs.brown.edu - ACL 2006 - 7.18.2006 - 12

  18. Summary of findings The self-trained WSJ + NANC model does not appear to be overtrained. Both self-training and reranking techniques are fairly portable across domains. WSJ data with these techniques gives performance almost as good as actual BROWN corpus (does not work as well with more distant domains) David McClosky - dmcc@cs.brown.edu - ACL 2006 - 7.18.2006 - 13

  19. Overview Introduction and Previous Work Parser portability Parser adaptation Reranker portability Analysis Future Work and Conclusions David McClosky - dmcc@cs.brown.edu - ACL 2006 - 7.18.2006 - 14

  20. Parser Portability Task: Use existing data/models from source domain to parse target domain. Train: WSJ Test: BROWN Variables: Parser vs. reranker parser Effect of self-training on NANC David McClosky - dmcc@cs.brown.edu - ACL 2006 - 7.18.2006 - 15

  21. Parser Portability Train Test Parser Reranking Parser 89.7 91.0 WSJ WSJ 83.9 85.8 WSJ BROWN f -score on WSJ section 23 and BROWN development section David McClosky - dmcc@cs.brown.edu - ACL 2006 - 7.18.2006 - 16

  22. Parser Portability Parsing model Parser Reranking Parser WSJ baseline 83.9 85.8 WSJ +50k NANC 84.8 86.6 WSJ +250k NANC 85.7 87.2 WSJ +500k NANC 86.0 87.3 WSJ +1,000k NANC 86.2 87.3 WSJ +1,500k NANC 86.2 87.6 WSJ +2,500k NANC 86.4 87.7 f -score on BROWN development section David McClosky - dmcc@cs.brown.edu - ACL 2006 - 7.18.2006 - 17

  23. Parser Portability Parsing model Parser Reranking Parser WSJ baseline 83.9 85.8 WSJ +50k NANC 84.8 86.6 WSJ +250k NANC 85.7 87.2 WSJ +500k NANC 86.0 87.3 WSJ +1,000k NANC 86.2 87.3 WSJ +1,500k NANC 86.2 87.6 WSJ +2,500k NANC 86.4 87.7 BROWN baseline 86.4 87.7 f -score on BROWN development section David McClosky - dmcc@cs.brown.edu - ACL 2006 - 7.18.2006 - 18

  24. Parser Adaptation Task: Use existing data/models from source domain with some target domain material to parse target domain. Train: WSJ and/or BROWN Test: BROWN Variables: Number of self-trained sentences added Amount of BROWN training data David McClosky - dmcc@cs.brown.edu - ACL 2006 - 7.18.2006 - 19

  25. Labeled In-domain Data Parser model Parser Reranker WSJ alone 83.9 85.8 BROWN alone 86.3 87.4 WSJ + BROWN 86.5 88.1 f -score on BROWN development section David McClosky - dmcc@cs.brown.edu - ACL 2006 - 7.18.2006 - 20

  26. Adding Self-Trained Data Parser model Parser Reranker WSJ alone 83.9 85.8 WSJ +2,500k NANC 86.4 87.7 BROWN alone 86.3 87.4 BROWN +250k NANC 86.8 88.1 WSJ + BROWN 86.5 88.1 WSJ + BROWN +250k NANC 86.8 88.1 f -score on BROWN development section David McClosky - dmcc@cs.brown.edu - ACL 2006 - 7.18.2006 - 21

  27. Reranker Portability Reranker Parser model Parser alone WSJ BROWN 82.9 85.2 85.2 WSJ WSJ + NANC 87.1 87.8 87.9 86.7 88.2 88.4 BROWN f -scores on BROWN test section David McClosky - dmcc@cs.brown.edu - ACL 2006 - 7.18.2006 - 22

  28. Reranker Portability Reranker Parser model Parser alone WSJ BROWN 82.9 85.2 85.2 WSJ WSJ + NANC 87.1 87.8 87.9 86.7 88.2 88.4 BROWN f -scores on BROWN test section David McClosky - dmcc@cs.brown.edu - ACL 2006 - 7.18.2006 - 22

  29. Reranker Portability Reranker Parser model Parser alone WSJ BROWN 82.9 85.2 85.2 WSJ WSJ + NANC 87.1 87.8 87.9 86.7 88.2 88.4 BROWN f -scores on BROWN test section David McClosky - dmcc@cs.brown.edu - ACL 2006 - 7.18.2006 - 22

  30. Analysis Overview Oracle scores Parser agreement Per-category f -scores Factor analysis David McClosky - dmcc@cs.brown.edu - ACL 2006 - 7.18.2006 - 23

  31. Oracle Scores Model 1-best 10-best 25-best 50-best 82.6 88.9 90.7 91.9 WSJ WSJ + NANC 86.4 92.1 93.5 94.3 86.3 92.0 93.3 94.2 BROWN f -score on BROWN development section David McClosky - dmcc@cs.brown.edu - ACL 2006 - 7.18.2006 - 24

  32. Oracle Scores Model 1-best 10-best 25-best 50-best 82.6 88.9 90.7 91.9 WSJ WSJ + NANC 86.4 92.1 93.5 94.3 86.3 92.0 93.3 94.2 BROWN f -score on BROWN development section David McClosky - dmcc@cs.brown.edu - ACL 2006 - 7.18.2006 - 24

  33. Parser Agreement Bracketing agreement f -score 88.03% Complete match 44.92% Average crossing brackets 0.94 POS Tagging agreement 94.85% Agreement of parses from WSJ + NANC reranking parser with parses from BROWN reranking parser David McClosky - dmcc@cs.brown.edu - ACL 2006 - 7.18.2006 - 25

  34. Per-Category f -scores Description Size WSJ + NANC ∆ BROWN Popular Lore 271 87.3 89.6 2.28 Letters 281 87.6 87.1 -0.45 General fiction 333 87.2 85.9 -1.29 Mystery 318 88.7 88.3 -0.45 Science fiction 76 87.7 88.8 1.17 Adventure 378 89.7 89.0 -0.64 Romance 338 88.0 86.6 -1.40 Humor 83 84.6 87.0 2.45 f -scores on BROWN development section David McClosky - dmcc@cs.brown.edu - ACL 2006 - 7.18.2006 - 26

  35. Factor Analysis Generalized linear model with binomial link with the predicted variable as BROWN f -score > WSJ + NANC f -score Explanatory variables: sentence length number of prepositions number of conjunctions BROWN subcorpus ID David McClosky - dmcc@cs.brown.edu - ACL 2006 - 7.18.2006 - 27

Download Presentation
Download Policy: The content available on the website is offered to you 'AS IS' for your personal information and use only. It cannot be commercialized, licensed, or distributed on other websites without prior consent from the author. To download a presentation, simply click this link. If you encounter any difficulties during the download process, it's possible that the publisher has removed the file from their server.

Recommend

HiddenVariable Models for Discriminative Reranking Terry Koo and Michael Collins {

HiddenVariable Models for Discriminative Reranking Terry Koo and Michael Collins {

HiddenVariable Models for Discriminative Reranking Terry Koo and Michael Collins { maestro|mcollins } @csail.mit.edu Overview of reranking The reranking approach Use a baseline model to get the N -best candidates Rerank the candidates

1.15k views • 53 slides
discrepancy for unsupervised domain adaptation Hongliang Yan 2017/06/21 Domain Adaptation DA

discrepancy for unsupervised domain adaptation Hongliang Yan 2017/06/21 Domain Adaptation DA

Mind the class weight bias: weighted maximum mean discrepancy for unsupervised domain adaptation Hongliang Yan 2017/06/21 Domain Adaptation DA Training Test Problem: (Source (Target) Training and test sets are related but under

495 views • 21 slides
Building a Predictive Parser I.e., How to build the parse table for a recursive-descent parser 1

Building a Predictive Parser I.e., How to build the parse table for a recursive-descent parser 1

Building a Predictive Parser I.e., How to build the parse table for a recursive-descent parser 1 Last Time: Intro LL(1) Predictive Parser Predict the parse tree top-down Parser structure 1 token of lookahead A stack tracking the

519 views • 37 slides
Parser Self-Training for Syntax-Based Machine Translation Nara Institute of Science and

Parser Self-Training for Syntax-Based Machine Translation Nara Institute of Science and

Parser Self-Training for Syntax-Based Machine Translation Nara Institute of Science and Technology Augmented Human Communication Laboratory Makoto Morishita, Koichi Akabe, Yuto Hatakoshi, Graham Neubig, Koichiro Yoshino, Satoshi

632 views • 37 slides
https://bazel.build/ Inputs /usr/bin/cc Action Outputs ./parser.h cc -I. -c parser.c -o

https://bazel.build/ Inputs /usr/bin/cc Action Outputs ./parser.h cc -I. -c parser.c -o

https://bazel.build/ Inputs /usr/bin/cc Action Outputs ./parser.h cc -I. -c parser.c -o parser.o ./parser.o ./parser.c Inputs /usr/bin/cc Action Outputs ./parser.h cc -I. -c parser.c -o parser.o ./parser.o ./parser.c Workspace

206 views • 19 slides
HMM-based acoustic model adaptation and discriminative training Steven Wegmann ICSI 11 April

HMM-based acoustic model adaptation and discriminative training Steven Wegmann ICSI 11 April

HMM-based acoustic model adaptation and discriminative training Steven Wegmann ICSI 11 April 2012 HMM-based adaptation and discriminative training are important techniques for improving accuracy Both procedures start with HMMs ML

745 views • 58 slides
Parser Larissa von Witte Institut fr Softwaretechnik und Programmiersprachen 11. Januar 2016

Parser Larissa von Witte Institut fr Softwaretechnik und Programmiersprachen 11. Januar 2016

Parser Larissa von Witte Institut fr Softwaretechnik und Programmiersprachen 11. Januar 2016 L. v. Witte 11. Januar 2016 1/23 Contents Introduction Taxonomy Recursive Descent Parser Shift Reduce Parser Parser Generators Parse Tree

638 views • 24 slides
MAP adaptation with SphinxTrain David Huggins-Daines dhuggins@cs.cmu.edu Language Technologies

MAP adaptation with SphinxTrain David Huggins-Daines dhuggins@cs.cmu.edu Language Technologies

MAP adaptation with SphinxTrain David Huggins-Daines dhuggins@cs.cmu.edu Language Technologies Institute Carnegie Mellon University MAP adaptation with SphinxTrain p.1/12 Theory of MAP adaptation Standard Baum-Welch training produces a

143 views • 12 slides
Neural Reranking Improves Subjective Quality of Machine Translation: NAIST at WAT 2015 Graham

Neural Reranking Improves Subjective Quality of Machine Translation: NAIST at WAT 2015 Graham

Neural Reranking Improves Subjective Quality of Machine Translation Neural Reranking Improves Subjective Quality of Machine Translation: NAIST at WAT 2015 Graham Neubig, Makoto Morishita, Satoshi Nakamura Nara Institute of Science and

713 views • 20 slides
Keep Calm Keep Calm and Use Parser and Use Parser Nov, 2015 Howard Huang, Huawei Julien

Keep Calm Keep Calm and Use Parser and Use Parser Nov, 2015 Howard Huang, Huawei Julien

Keep Calm Keep Calm and Use Parser and Use Parser Nov, 2015 Howard Huang, Huawei Julien Zhang, ZTE How would OPNFV attract more open source software developers ? 2 I wrote the script based on the instructions in ETSI NFV documents, but

293 views • 15 slides
WSD for n -best reranking and local language modeling in SMT Marianna Apidianaki, Guillaume

WSD for n -best reranking and local language modeling in SMT Marianna Apidianaki, Guillaume

WSD for n -best reranking and local language modeling in SMT Marianna Apidianaki, Guillaume Wisniewski, Artem Sokolov, Aurlien Max, Franois Yvon LIMSI-CNRS & Univ. Paris Sud Orsay, France Sixth Workshop on Syntax, Semantics and

459 views • 29 slides
Combining Active Learning and Partial Annotation for Domain Adaptation of a Japanese Dependency

Combining Active Learning and Partial Annotation for Domain Adaptation of a Japanese Dependency

Combining Active Learning and Partial Annotation for Domain Adaptation of a Japanese Dependency Parser Daniel FLANNERY 1 Shinsuke MORI 2 1 Vitei Inc. (work at Kyoto University) 2 Kyoto University IWPT 2015, July 22nd 1 / 29 IWPT95 at Prague

571 views • 29 slides
Coastal Adaptation Kellie Fisher FCERM Senior Advisor Why Adaptation? Adaptation to a

Coastal Adaptation Kellie Fisher FCERM Senior Advisor Why Adaptation? Adaptation to a

Coastal Adaptation Kellie Fisher FCERM Senior Advisor Why Adaptation? Adaptation to a changing environment. Coastal change is not new! Traditional defences are a form of adaptation too. Are they always appropriate? Property

516 views • 19 slides
Tasks of a Parser Tasks of a Parser Document Parser Interfaces Document Parser Interfaces

Tasks of a Parser Tasks of a Parser Document Parser Interfaces Document Parser Interfaces

2. XML Processor APIs Document Parser Interfaces 2. XML Processor APIs Document Parser Interfaces (See, e.g., Leventhal (See, e.g., Leventhal, Lewis & Fuchs: Designing XML , Lewis & Fuchs: Designing XML How can (Java) applications

746 views • 4 slides
Saving Data in iOS Using XML NSXMLParser It is the default XML parser included in iOS It is a

Saving Data in iOS Using XML NSXMLParser It is the default XML parser included in iOS It is a

Saving Data in iOS Using XML NSXMLParser It is the default XML parser included in iOS It is a SAX (Simple API for XML) based XML parser. It is fast and must be tuned for the XML feed that you are parsing. Uses a delegate to handle events as

698 views • 14 slides
A Transition-Based Directed Acyclic Graph Parser for Universal Conceptual Cognitive Annotation

A Transition-Based Directed Acyclic Graph Parser for Universal Conceptual Cognitive Annotation

1 A Transition-Based Directed Acyclic Graph Parser for Universal Conceptual Cognitive Annotation Daniel Hershcovich, Omri Abend and Ari Rappoport ACL 2017 2 TUPA Transition-based UCCA Parser The first parser to support the combination of

872 views • 71 slides
Social Networking with Frientegrity: Privacy and Integrity with an Untrusted Provider Ariel

Social Networking with Frientegrity: Privacy and Integrity with an Untrusted Provider Ariel

Social Networking with Frientegrity: Privacy and Integrity with an Untrusted Provider Ariel J. Feldman Princeton UPenn Joint work with: Aaron Blankstein, Michael J. Freedman, and Edward W. Felten Social Networking with

661 views • 27 slides
Building an Aragon App An Aragon App uses standard interfaces to support governance and

Building an Aragon App An Aragon App uses standard interfaces to support governance and

Building an Aragon App An Aragon App uses standard interfaces to support governance and upgradeability Governance as an interface: Defines how entities can perform actions traverse ACL DAO ACL permissions exec team , vault, transfer, ANT

337 views • 15 slides
Align, Disambiguate, and Walk A Unified Approach for Measuring Semantic Similarity Semantic

Align, Disambiguate, and Walk A Unified Approach for Measuring Semantic Similarity Semantic

Align, Disambiguate, and Walk A Unified Approach for Measuring Semantic Similarity Semantic Similarity; how similar are a pair of lexical items? Semantic Similarity Semantic Similarity Semantic Similarity Sentence level

1.42k views • 127 slides
How to Read/Write an International Conference Paper Graham Neubig Nara Institute of Science and

How to Read/Write an International Conference Paper Graham Neubig Nara Institute of Science and

H o w t o R e a d / Wr i t e a n I n t e r n a t i o n a l C o n f e r e n c e P a p e r How to Read/Write an International Conference Paper Graham Neubig Nara Institute of Science and Technology

1.03k views • 88 slides
Rogue Femtocell Owners: How Mallory Can Monitor My Devices David Malone, Darren F. Kavanagh and

Rogue Femtocell Owners: How Mallory Can Monitor My Devices David Malone, Darren F. Kavanagh and

Rogue Femtocell Owners: How Mallory Can Monitor My Devices David Malone, Darren F. Kavanagh and Niall R. Murphy 19 April 2013 Femtocells Small devices acting as cellular base stations. Deployed to extend coverage in homes, offices, . . .

450 views • 20 slides
A Low-budget Tagger for Old Czech Jirka Hana 1 Anna Feldman 2 Katsiaryna Aharodnik 2 1 Charles

A Low-budget Tagger for Old Czech Jirka Hana 1 Anna Feldman 2 Katsiaryna Aharodnik 2 1 Charles

A Low-budget Tagger for Old Czech Jirka Hana 1 Anna Feldman 2 Katsiaryna Aharodnik 2 1 Charles University, Prague 2 Montclair State University, NJ ACL 2011 LaTeCH Portland, OR, June 24, 2010 J. Hana et al. (Charles University & MSU) A

696 views • 41 slides
[show me your privileges and I will lead you to SYSTEM] Andrea Pierini, Paris, June 19 th 2019 1

[show me your privileges and I will lead you to SYSTEM] Andrea Pierini, Paris, June 19 th 2019 1

[show me your privileges and I will lead you to SYSTEM] Andrea Pierini, Paris, June 19 th 2019 1 dir /a /r %USERPROFILE% Cyclist & Scuba Diver, Father & Husband IT Architect & Security Manager Long time experience

901 views • 62 slides
Sparsity in Dependency Grammar Induction Jennifer Gillenwater 1 Kuzman Ganchev 1 ca 2 Jo ao

Sparsity in Dependency Grammar Induction Jennifer Gillenwater 1 Kuzman Ganchev 1 ca 2 Jo ao

Sparsity in Dependency Grammar Induction Jennifer Gillenwater 1 Kuzman Ganchev 1 ca 2 Jo ao Gra Ben Taskar 1 Fernando Pereira 3 1 Computer & Information Science University of Pennsylvania 2 L 2 F INESC-ID, Lisboa, Portugal 3 Google, Inc.

354 views • 34 slides

More recommend