hidden variable models for discriminative reranking
play

HiddenVariable Models for Discriminative Reranking Terry Koo and - PowerPoint PPT Presentation

Jul 04, 2023 •604 likes •1.15k views

HiddenVariable Models for Discriminative Reranking Terry Koo and Michael Collins { maestro|mcollins } @csail.mit.edu Overview of reranking The reranking approach Use a baseline model to get the N -best candidates Rerank the candidates

  1. Hidden–Variable Models for Discriminative Reranking Terry Koo and Michael Collins { maestro|mcollins } @csail.mit.edu

  2. Overview of reranking The reranking approach Use a baseline model to get the N -best candidates “Rerank” the candidates using a more complex model Parse reranking Collins (2000): 88.2% ⇒ 89.8% Charniak and Johnson (2005): 89.7% ⇒ 91.0% Talk by Brooke Cowan in 7B: 83.6% ⇒ 85.1% Also applied to MT (Och and Ney, 2002; Shen et al., 2004) NL Generation (Walker et al., 2001)

  3. Representing NLP structures Proper representation is critical to success Hand–crafted feature vector representations Φ ( ) = { 0, 1, 2, 0, 0, 3, 0, 1 } Features defined through kernels K ( ) · Φ ( , ) = Φ ( ) This talk: A new approach using hidden variables

  4. Two facets of lexical items Different lexical items can have similar meanings, e.g. president and chairman Clustering: president , chairman ∈ NounCluster 4 A single lexical item can have different meanings, e.g. [river] bank vs [financial] bank Refinement: bank 1 , bank 2 ∈ bank Model clusterings and refinements as hidden variables that support the reranking task

  5. Highlights of the approach Conditional log–linear model with hidden variables Dynamic programming is used for training and decoding Clustering and refinement done automatically using a discriminative criterion

  6. Overview of talk Motivation Design General form of the model Training and decoding efficiently Creating specific instantiations Results Discussion Conclusion

  7. The parse reranking framework Sentences s i for 1 ≤ i ≤ n s 1 : Pierre Vinken , 61 years old , will join ... s 2 : Mr. Vinken is chairman of Elsevier N.V. ... s 3 : Big Board Chairman John Phelan said yesterday ... Each s i has candidate parses t i , j for 1 ≤ j ≤ n i t i ,1 is the best candidate parse for s i

  8. The parse reranking framework t i , j has phrase structure and dependency tree S VP NP PP NP NP NP NNP NNP VB NN IN NNP NNP Mr. Vinken is chairman of Elsevier N.V.

  9. The parse reranking framework t i , j has phrase structure and dependency tree S VP NP PP NP NP NP NNP NNP VB NN IN NNP NNP Mr. Vinken is chairman of Elsevier N.V.

  10. Adding hidden variables Hidden–value domains H w ( t i , j ) for 1 ≤ w ≤ len( s i ) S VP NP PP NP NP NP NNP NNP VB NN IN NNP NNP Mr. Vinken is chairman of Elsevier N.V. NNP NNP VB 1 NN IN 1 NNP NNP 1 1 1 1 1 NNP NNP VB NN IN 2 NNP NNP 2 2 2 2 2 2 NNP NNP VB NN 3 IN 3 NNP NNP 3 3 3 3 3

  11. Adding hidden variables Assignment h ∈ H 1 ( t i , j ) × ... × H len( s i ) ( t i , j ) S VP NP PP NP NP NP NNP NNP VB NN IN NNP NNP Mr. Vinken is chairman of Elsevier N.V. NNP NNP VB 1 NN IN 1 NNP NNP 1 1 1 1 1 NNP NNP VB NN IN 2 NNP NNP 2 2 2 2 2 2 NNP NNP VB NN 3 IN 3 NNP NNP 3 3 3 3 3

  12. Marginalized probability model Φ ( t i , j , h ) produces a descriptive vector of feature occurrence counts, e.g. Φ 2 ( t i , j , h ) = Count( chairman has hidden value NN 1 ) Φ 13 ( t i , j , h ) = Count(NNP 2 is a direct object of VB 1 ) Φ 19 ( t i , j , h ) = Count(NN 1 coordinates with NN 2 )

  13. Marginalized probability model Log–linear distribution over ( t i , j , h ) with parameters Θ : e Φ ( t i , j , h ) · Θ p ( t i , j , h | s i , Θ ) = j ′ , h ′ e Φ ( t i , j ′ , h ′ ) · Θ � Marginalize over assignments h : p ( t i , j | s i , Θ ) = � p ( t i , j , h | s i , Θ ) h

  14. Optimizing the parameters Define loss as negative log-likelihood n L ( Θ ) = - log p ( t i ,1 | s i , Θ ) � i=1 Minimize L ( Θ ) through gradient descent p ( h | t i ,1 , s i , Θ ) Φ ( t i ,1 , h ) ∂ L ∂ Θ = - � � i h p ( t i , j | s i , Θ ) � p ( h | t i , j , s i , Θ ) Φ ( t i , j , h ) + � i , j h

  15. Overview of talk Motivation Design General form of the model Training and decoding efficiently Creating specific instantiations Results Discussion Conclusion

  16. Problems with efficiency | H 1 ( t i , j ) × ... × H len( s i ) ( t i , j ) | grows exponentially, so training the model is intractable: p ( h | t i ,1 , s i , Θ ) Φ ( t i ,1 , h ) ∂ L ∂ Θ = - � � h i p ( t i , j | s i , Θ ) � p ( h | t i , j , s i , Θ ) Φ ( t i , j , h ) + � h i , j Decoding the model is also intractable: p ( t i , j | s i , Θ ) = � p ( t i , j , h | s i , Θ ) h

  17. Problems with efficiency | H 1 ( t i , j ) × ... × H len( s i ) ( t i , j ) | grows exponentially, so training the model is intractable: p ( h | t i ,1 , s i , Θ ) Φ ( t i ,1 , h ) ∂ L ∂ Θ = - � � h i p ( t i , j | s i , Θ ) � p ( h | t i , j , s i , Θ ) Φ ( t i , j , h ) + � h i , j Decoding the model is also intractable: p ( t i , j | s i , Θ ) = � p ( t i , j , h | s i , Θ ) h

  18. Locality constraint on features Features have pairwise local scope on hidden variables Features still have global scope on non-hidden information Φ can be factored into local feature vectors, allowing dynamic programming

  19. Local feature vectors Define two kinds of local feature vector φ : Single-variable φ ( t i , j , w , h w ) look at a single hidden variable Pairwise φ ( t i , j , u , v , h u , h v ) look at two hidden variables in a dependency relationship

  20. Local feature vectors Φ ( t i , j , h ) looks at every hidden variable S VP NP PP NP NP NP NNP NNP VB NN IN NNP NNP Mr. Vinken is chairman of Elsevier N.V. NNP NNP VB 1 NN IN 1 NNP NNP 1 1 1 1 1 NNP NNP VB NN IN 2 NNP NNP 2 2 2 2 2 2 NNP NNP VB NN 3 IN 3 NNP NNP 3 3 3 3 3

  21. Local feature vectors φ ( t i , j , chairman , NN 3 ) only sees NN 3 S VP NP PP NP NP NP NNP NNP VB NN IN NNP NNP Mr. Vinken is chairman of Elsevier N.V. NN 1 NN 2 NN 3

  22. Local feature vectors φ ( t i , j , chairman , of , NN 3 , IN 2 ) sees NN 3 and IN 2 S VP NP PP NP NP NP NNP NNP VB NN IN NNP NNP Mr. Vinken is chairman of Elsevier N.V. NN IN 1 1 NN IN 2 2 NN 3 IN 3

  23. Local feature vectors Rewrite global Φ as a sum over local φ Φ ( t i , j , h ) = φ ( t i , j , w , h w ) � w ∈ ti , j φ ( t i , j , u , v , h u , h v ) + � ( u , v ) ∈ D ( ti , j )

  24. Local feature vectors Rewrite global Φ as a sum over local φ Φ ( t i , j , h ) = φ ( t i , j , w , h w ) � w ∈ ti , j φ ( t i , j , u , v , h u , h v ) + � ( u , v ) ∈ D ( ti , j )

  25. Applying belief propagation New restrictions enable dynamic–programming approaches, e.g. belief propagation BP generalizes the forward–backward algorithm from a chain to a tree Runtime O (len( s i ) H 2 ), H = max | H w ( t i , j ) | BP efficiently computes p ( t i , j , h | s i , Θ ) � h p ( h | t i , j , s i , Θ ) Φ ( t i , j , h ) � h

  26. Overview of talk Motivation Design General form of the model Training and decoding efficiently Creating specific instantiations Results Discussion Conclusion

  27. Two areas for choice in the model Definition of the hidden–value domains H w ( t i , j ) Definition of the feature vectors φ

  28. Hidden–value domains Lexical domains allow word refinement Mr. Vinken is chairman of 1 Elsevier N.V. 1 1 1 1 1 1 Mr. Vinken is chairman of Elsevier N.V. 2 2 2 2 2 2 2 Mr. Vinken is chairman of Elsevier N.V. 3 3 3 3 3 3 3 Mr. Vinken is chairman of Elsevier N.V.

  29. Hidden–value domains Lexical domains allow word refinement Mr. Vinken is chairman 1 of 1 Elsevier N.V. 1 1 1 1 1 Mr. Vinken is chairman of Elsevier N.V. 2 2 2 2 2 2 2 Mr. Vinken is chairman of Elsevier N.V. 3 3 3 3 3 3 3 Mr. Vinken is chairman of Elsevier N.V.

  30. Hidden–value domains Part-of-speech domains allow word clustering NNP NNP VB NN 1 IN NNP NNP 1 1 1 1 1 1 NNP NNP VB NN IN NNP NNP 2 2 2 2 2 2 2 NNP NNP VB NN IN NNP NNP 3 3 3 3 3 3 3 NNP NNP VB NN IN NNP NNP 4 4 4 4 4 4 4 NNP NNP VB NN IN NNP NNP 5 5 5 5 5 5 5 Mr. Vinken is chairman of Elsevier N.V.

  31. Hidden–value domains Part-of-speech domains allow word clustering NNP NNP VB NN 1 IN NNP NNP 1 1 1 1 1 1 NNP NNP VB NN IN NNP NNP 2 2 2 2 2 2 2 NNP NNP VB NN IN NNP NNP 3 3 3 3 3 3 3 NNP NNP VB NN IN NNP NNP 4 4 4 4 4 4 4 NNP NNP VB NN IN NNP NNP 5 5 5 5 5 5 5 Mr. Vinken is chairman of Elsevier N.V.

  32. Hidden–value domains Part-of-speech domains allow word clustering NNP NNP VB NN 1 IN NNP NNP 1 1 1 1 1 1 NNP NNP VB NN IN NNP NNP 2 2 2 2 2 2 2 NNP NNP VB NN IN NNP NNP 3 3 3 3 3 3 3 NNP NNP VB NN IN NNP NNP 4 4 4 4 4 4 4 NNP NNP VB NN IN NNP NNP 5 5 5 5 5 5 5 Mr. Vinken is chairman of Elsevier N.V.

  33. Hidden–value domains Supersense domains model WordNet ontology (Ciaramita and Johnson, 2003; Miller et at., 1993) NNP NNP verb.stative noun.person IN NNP NNP 1 1 1 1 1 1 1 NNP NNP verb.stative noun.person IN NNP NNP 2 2 2 2 2 2 2 NNP NNP verb.stative noun.person IN NNP NNP 3 3 3 3 3 3 3 NNP NNP verb.social IN NNP NNP 4 4 1 4 4 4 chairman NNP NNP verb.social IN NNP NNP 5 5 2 5 5 5 verb.social Mr. Vinken of Elsevier N.V. 3 verb.possession 1 verb.possession 2 verb.possession 3 is

Download Presentation
Download Policy: The content available on the website is offered to you 'AS IS' for your personal information and use only. It cannot be commercialized, licensed, or distributed on other websites without prior consent from the author. To download a presentation, simply click this link. If you encounter any difficulties during the download process, it's possible that the publisher has removed the file from their server.

Recommend

z A single Gaussian might be a poor fit . . . . . Simplest form is 2 layer . . . ... .

z A single Gaussian might be a poor fit . . . . . Simplest form is 2 layer . . . ... .

Overview Hidden Variable Models 1: Mixture Models Hidden variable models Mixture models Chris Williams Mixtures of Gaussians Aside: Kullback-Leibler divergence School of Informatics, University of Edinburgh The EM algorithm October 2008

395 views • 6 slides
Discriminative Regularization for Latent Variable Models with Applications to Electrocardiography

Discriminative Regularization for Latent Variable Models with Applications to Electrocardiography

Discriminative Regularization for Latent Variable Models with Applications to Electrocardiography Poster #53 Andrew C. Miller with Ziad Obermeyer, John P . Cunningham, and Sendhil Mullainathan <latexit

674 views • 9 slides
Concept-to-text Generation via Discriminative Reranking Ioannis Konstas and Mirella Lapata School

Concept-to-text Generation via Discriminative Reranking Ioannis Konstas and Mirella Lapata School

Concept-to-text Generation via Discriminative Reranking Ioannis Konstas and Mirella Lapata School of Informatics Institute for Language, Cognition and Computation University of Edinburgh ACL 2012, Jeju Island Konstas, Lapata (ILCC)

1.17k views • 69 slides
Discriminative Models Joakim Nivre Uppsala University Department of Linguistics and Philology

Discriminative Models Joakim Nivre Uppsala University Department of Linguistics and Philology

Discriminative Models Joakim Nivre Uppsala University Department of Linguistics and Philology joakim.nivre@lingfil.uu.se Discriminative Models 1(11) 1. Generative and Discriminative Models 2. Log-Linear Models 3. Local Discriminative Models

278 views • 11 slides
Learning Overcomplete Latent Variable Models through Tensor Methods Anima Anandkumar UC Irvine

Learning Overcomplete Latent Variable Models through Tensor Methods Anima Anandkumar UC Irvine

Learning Overcomplete Latent Variable Models through Tensor Methods Anima Anandkumar UC Irvine Joint work with Majid Janzamin Rong Ge UC Irvine Microsoft Research Latent Variable Probabilistic Models Latent (hidden) variable h R k ,

769 views • 47 slides
Learning Latent Variable Models through Tensor Methods Anima Anandkumar U.C. Irvine Challenges

Learning Latent Variable Models through Tensor Methods Anima Anandkumar U.C. Irvine Challenges

Learning Latent Variable Models through Tensor Methods Anima Anandkumar U.C. Irvine Challenges in Unsupervised Learning Learn a latent variable model without labeled examples. E.g. topic models, hidden Markov models, Gaussian mixtures,

763 views • 61 slides
Language and Document Analysis: Motivating Latent variable Models Wray Buntine National ICT

Language and Document Analysis: Motivating Latent variable Models Wray Buntine National ICT

Language and Document Analysis: Motivating Latent variable Models Wray Buntine National ICT Australia (NICTA) MLSS, ANU, Jan., 2009 Buntine Document Models Part-of-Speech with Hidden Markov Models Topics in Text with Discrete Component

1.25k views • 59 slides
Learning Overcomplete Latent Variable Models through Tensor Methods Majid Janzamin UC Irvine

Learning Overcomplete Latent Variable Models through Tensor Methods Majid Janzamin UC Irvine

Learning Overcomplete Latent Variable Models through Tensor Methods Majid Janzamin UC Irvine Joint work with Anima Anandkumar Rong Ge UC Irvine Microsoft Research Latent Variable Modeling Goal: Discover hidden effects from observed

1.27k views • 115 slides
1 Latent variable models In the next section we will discuss latent variable models for

1 Latent variable models In the next section we will discuss latent variable models for

Statistical Modeling and Analysis of Neural Data (NEU 560) Princeton University, Spring 2018 Jonathan Pillow Lecture 16 notes: Latent variable models and EM Tues, 4.10 1 Latent variable models In the next section we will discuss latent

271 views • 4 slides
Random Variable : non empty set. {up, town} Event A: is a subset of . {up} A

Random Variable : non empty set. {up, town} Event A: is a subset of . {up} A

Introduction to Probability and Hidden Markov Models Durbin et al. Ch3 EECS 458 CWRU Fall 2004 Random Variable : non empty set. {up, town} Event A: is a subset of . {up} A random variable X is a function defined on events

388 views • 36 slides
Three models for discriminative machine Three models for discriminative machine translation using

Three models for discriminative machine Three models for discriminative machine translation using

Three models for discriminative machine Three models for discriminative machine translation using Global Lexical Selection translation using Global Lexical Selection and Sentence Reconstruction and Sentence Reconstruction Sriram Venkatapathy

645 views • 32 slides
A Discriminative Latent Variable Model for Online Clustering Rajhans Samdani, Kai-Wei Chang , Dan

A Discriminative Latent Variable Model for Online Clustering Rajhans Samdani, Kai-Wei Chang , Dan

A Discriminative Latent Variable Model for Online Clustering Rajhans Samdani, Kai-Wei Chang , Dan Roth Department of Computer Science University of Illinois at Urbana-Champaign Motivating Example: Coreference n Coreference resolution: cluster

510 views • 22 slides
CSE P 590 A Markov Models and Hidden Markov Models

CSE P 590 A Markov Models and Hidden Markov Models

CSE P 590 A Markov Models and Hidden Markov Models http://upload.wikimedia.org/wikipedia/commons/b/ba/Calico_cat Dosage Compensation Reminder: Proteins Read DNA and X-Inactivation E.g.: 2 copies (mom/dad) of each

1.28k views • 21 slides
1 X 1 X 2 X 3 Ghostbusters HMM Chain Rule and HMMs E 1 E 2 E 3 P(X 1 ) = uniform 1/9 1/9

1 X 1 X 2 X 3 Ghostbusters HMM Chain Rule and HMMs E 1 E 2 E 3 P(X 1 ) = uniform 1/9 1/9

Hidden Markov Models 1 Hidden Markov Models Markov chains not so useful for most agents Need observations to update your beliefs Hidden Markov models (HMMs) Underlying Markov chain over states X You observe outputs (effects) at

617 views • 7 slides
Discriminative Metric Learning in Nearest Neighbor Models for Image Annotation Matthieu

Discriminative Metric Learning in Nearest Neighbor Models for Image Annotation Matthieu

Discriminative Metric Learning in Nearest Neighbor Models for Image Annotation Matthieu Guillaumin, Thomas Mensink, Jakob Verbeek & Cordelia Schmid LEAR Team, INRIA Rh one-Alpes Grenoble, France Discriminative Metric Learning in Nearest

555 views • 40 slides
1 Real HMM Examples Real HMM Examples Speech recognition HMMs: Machine translation HMMs:

1 Real HMM Examples Real HMM Examples Speech recognition HMMs: Machine translation HMMs:

Hidden Markov Models CSE 473: Artificial Intelligence Markov chains not so useful for most agents Hidden Markov Models Eventually you dont know anything anymore Need observations to update your beliefs Hidden Markov models

567 views • 9 slides
Part-of-Speech Tagging COSI 114 Computational Linguistics James Pustejovsky March 17, 2017

Part-of-Speech Tagging COSI 114 Computational Linguistics James Pustejovsky March 17, 2017

Part-of-Speech Tagging COSI 114 Computational Linguistics James Pustejovsky March 17, 2017 Brandeis University Parts of Speech Perhaps starting with Aristotle in the West (384322 BCE) the idea of having parts of speech lexical

554 views • 54 slides
Syntax-Based Decoding Philipp Koehn 9 November 2017 Philipp Koehn Machine Translation:

Syntax-Based Decoding Philipp Koehn 9 November 2017 Philipp Koehn Machine Translation:

Syntax-Based Decoding Philipp Koehn 9 November 2017 Philipp Koehn Machine Translation: Syntax-Based Decoding 9 November 2017 1 syntax-based models Philipp Koehn Machine Translation: Syntax-Based Decoding 9 November 2017 Synchronous

740 views • 61 slides
Information Extraction Prof. Sameer Singh CS 295: STATISTICAL NLP WINTER 2017 February 21, 2017

Information Extraction Prof. Sameer Singh CS 295: STATISTICAL NLP WINTER 2017 February 21, 2017

Information Extraction Prof. Sameer Singh CS 295: STATISTICAL NLP WINTER 2017 February 21, 2017 Based on slides from Dan Jurafski, Chris Manning, Jay Pujara, and everyone else they copied from. Outline What is Information Extraction Named

896 views • 35 slides
Named Entity Recognition Katharine Jarmul Founder, kjamistan DataCamp Introduction to Natural

Named Entity Recognition Katharine Jarmul Founder, kjamistan DataCamp Introduction to Natural

DataCamp Introduction to Natural Language Processing in Python INTRODUCTION TO NATURAL LANGUAGE PROCESSING IN PYTHON Named Entity Recognition Katharine Jarmul Founder, kjamistan DataCamp Introduction to Natural Language Processing in Python

600 views • 17 slides
Treebank Grammars and Parser Evaluation Syntactic analysis (5LN455) 2016-11-15 Sara Stymne

Treebank Grammars and Parser Evaluation Syntactic analysis (5LN455) 2016-11-15 Sara Stymne

Treebank Grammars and Parser Evaluation Syntactic analysis (5LN455) 2016-11-15 Sara Stymne Department of Linguistics and Philology Based on slides from Marco Kuhlmann Recap: Probabilistic parsing Probabilistic context-free grammars A

802 views • 42 slides
Radiative pion capture in 2 H, 3 He and 3 H J. Golak , R. Skibiski, K. Topolnicki, H. Witaa,

Radiative pion capture in 2 H, 3 He and 3 H J. Golak , R. Skibiski, K. Topolnicki, H. Witaa,

Radiative pion capture in 2 H, 3 He and 3 H J. Golak , R. Skibiski, K. Topolnicki, H. Witaa, A. Grassi, H. Kamada, A. Nogga, L.E. Marcucci JAGIELLONIAN UNIVERSITY 15th International Workshop on Meson Physics KRAKW, POLAND , 7th - 12th

630 views • 37 slides
Filtering relevant information from reports on flood Lubo s Popel nsk y Knowledge

Filtering relevant information from reports on flood Lubo s Popel nsk y Knowledge

Filtering relevant information from reports on flood Lubo s Popel nsk y Knowledge Discovery Lab Faculty of Informatics, Masaryk University, Brno, Czech Republic Example: News report on flood news reports on flood (period of 2002 in

274 views • 12 slides
? (entity type) Apr 23, 2007 NAACL-HLT 2 1 What Is Relation Extraction? hundreds of

? (entity type) Apr 23, 2007 NAACL-HLT 2 1 What Is Relation Extraction? hundreds of

A Systematic Exploration of the Feature Space for Relation Extraction Jing Jiang & ChengXiang Zhai Department of Computer Science University of Illinois, Urbana-Champaign What Is Relation Extraction? hundreds of Palestinians converged

678 views • 23 slides

More recommend