syntactically guided neural machine translation
play

Syntactically Guided Neural Machine Translation Felix Stahlberg, - PowerPoint PPT Presentation

Mar 11, 2023 •1.48k likes •1.71k views

Syntactically Guided Neural Machine Translation Felix Stahlberg, Eva Hasler, Aurelien Waite, and Bill Byrne Department of Engineering Syntactically Guided Neural Machine Translation Felix Stahlberg, Eva Hasler, Aurelien Waite, and Bill Byrne

  1. Syntactically Guided Neural Machine Translation Felix Stahlberg, Eva Hasler, Aurelien Waite, and Bill Byrne Department of Engineering Syntactically Guided Neural Machine Translation Felix Stahlberg, Eva Hasler, Aurelien Waite, and Bill Byrne

  2. Neural machine translation (NMT) vs. Hiero NMT Hiero • Simple beam search* • Searches over a vast number of translations • No explicit coverage • CKY parses cover the mechanism* complete source sentence • Limited vocabulary size* • Very large vocabularies, open to extension • Long-range context (RNN) • Limited LM context, weak translation model *: Vanilla formulation of attentional NMT according Bahdanau et al., 2015 Syntactically Guided Neural Machine Translation Felix Stahlberg, Eva Hasler, Aurelien Waite, and Bill Byrne

  3. Combining NMT and Hiero scores x : Source sentence • NMT left-to-right factorization: 𝑈 : Target sentence y = 𝑧 1 • NMT+Hiero via log-linear model combination Hiero predictive posteriors through FST weight pushing UNK score is used for NMT OOVs Syntactically Guided Neural Machine Translation Felix Stahlberg, Eva Hasler, Aurelien Waite, and Bill Byrne

  4. FST weight pushing Hiero lattice: <s>|0.5 </s>|0.1 b|0.5 a|0.4 a|0.8 c|0.4 </s>|1.0 b|0.5 b|0.8 c|0.9 Hiero lattice after weight pushing: <s>|0.409 b|0.061 </s>|1.0 a|0.042 a|0.042 a|0.939 c|0.208 c|0.208 </s>|1.0 b|1.0 b|0.75 b|0.75 c|1.0 𝑄 𝐼𝑗𝑓𝑠𝑝 𝑧 3 = a s a, 𝐲 = 0.042 𝑄 𝐼𝑗𝑓𝑠𝑝 𝑧 3 = c s a, 𝐲 = 0.208 𝑄 𝐼𝑗𝑓𝑠𝑝 𝑧 3 = b s a, 𝐲 = 0.75 Syntactically Guided Neural Machine Translation Felix Stahlberg, Eva Hasler, Aurelien Waite, and Bill Byrne

  5. Results on news-test2014 English-German English-French (BLEU) (BLEU) Baselines and related work Hiero baseline (de Gispert et al., 2010) 19.44 32.86 Basic NMT (RNNsearch) (Bahdanau et al., 2015) 16.31 30.42 RNNsearch-LV + UNK Replace (Jean et al., 2015) 19.40 34.60 This work Syntactically guided NMT ( 𝜇 𝐼𝑗𝑓𝑠𝑝 = 0 ) 20.69 35.37 Syntactically guided NMT (tuned 𝜇 𝑂𝑁𝑈 , 𝜇 𝐼𝑗𝑓𝑠𝑝 ) 21.87 36.61 Syntactically Guided Neural Machine Translation Felix Stahlberg, Eva Hasler, Aurelien Waite, and Bill Byrne

  6. Results on news-test2015 (English-German) Search space # of node expansions BLEU per sentence 100-best rescoring 2,233.6 22.9 (Depth-First Search: 832.1) 1000-best rescoring 21,686.2 23.5 (Depth-First Search: 6,221.8) Lattice-based 244.3 24.0 (Syntactically guided NMT) NMT baseline: 19.5 BLEU Hiero baseline (with NPLM): 21.7 BLEU Syntactically Guided Neural Machine Translation Felix Stahlberg, Eva Hasler, Aurelien Waite, and Bill Byrne

  7. Conclusion • Using syntactic SMT to guide neural machine translation yields great potential • Our lattice-based approach is faster and better than n-best list rescoring • More discussion in the paper • NMT modelling vs. search errors • Local softmax • Beam size • Lattice size • … Syntactically Guided Neural Machine Translation Felix Stahlberg, Eva Hasler, Aurelien Waite, and Bill Byrne

  8. References • Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. Neural machine translation by jointly learning to align and translate. In ICLR • Adria de Gispert, Gonzalo Iglesias, Graeme Blackwood, Eduardo R Banga, and William Byrne. 2010. Hierarchical phrase-based translation with weighted finite-state transducers and shallow-n grammars. Computational Linguistics, 36(3):505 – 533. • Sebastien Jean, Kyunghyun Cho, Roland Memisevic, and Yoshua Bengio. 2015a. On using very large target vocabulary for neural machine translation. In ACL, pages 1 – 10. Syntactically Guided Neural Machine Translation Felix Stahlberg, Eva Hasler, Aurelien Waite, and Bill Byrne

  9. Thanks Code available at http://ucam-smt.github.io/sgnmt/html Syntactically Guided Neural Machine Translation Felix Stahlberg, Eva Hasler, Aurelien Waite, and Bill Byrne

  10. BACKUP Syntactically Guided Neural Machine Translation Felix Stahlberg, Eva Hasler, Aurelien Waite, and Bill Byrne

  11. Results Syntactically Guided Neural Machine Translation Felix Stahlberg, Eva Hasler, Aurelien Waite, and Bill Byrne

  12. Beam size Syntactically Guided Neural Machine Translation Felix Stahlberg, Eva Hasler, Aurelien Waite, and Bill Byrne

  13. Lattice size Syntactically Guided Neural Machine Translation Felix Stahlberg, Eva Hasler, Aurelien Waite, and Bill Byrne

  14. Data Syntactically Guided Neural Machine Translation Felix Stahlberg, Eva Hasler, Aurelien Waite, and Bill Byrne

  15. RNN Update Syntactically Guided Neural Machine Translation Felix Stahlberg, Eva Hasler, Aurelien Waite, and Bill Byrne

Download Presentation
Download Policy: The content available on the website is offered to you 'AS IS' for your personal information and use only. It cannot be commercialized, licensed, or distributed on other websites without prior consent from the author. To download a presentation, simply click this link. If you encounter any difficulties during the download process, it's possible that the publisher has removed the file from their server.

Recommend

Neural Machine Translation Gongbo Tang 8 October 2018 Outline Neural Machine Translation 1

Neural Machine Translation Gongbo Tang 8 October 2018 Outline Neural Machine Translation 1

Neural Machine Translation Gongbo Tang 8 October 2018 Outline Neural Machine Translation 1 Advances and Challenges 2 Gongbo Tang Neural Machine Translation 2/52 Neural Machine Translation Figure Recurrent neural network based NMT

906 views • 73 slides
Neural Machine Translation Philipp Koehn 6 October 2020 Philipp Koehn Machine Translation:

Neural Machine Translation Philipp Koehn 6 October 2020 Philipp Koehn Machine Translation:

Neural Machine Translation Philipp Koehn 6 October 2020 Philipp Koehn Machine Translation: Neural Machine Translation 6 October 2020 Language Models 1 Modeling variants feed-forward neural network recurrent neural network long

485 views • 36 slides
Convolutional over Recurrent Encoder for Neural Machine Translation Praveen Dakwale and Christof

Convolutional over Recurrent Encoder for Neural Machine Translation Praveen Dakwale and Christof

Annual Conference of the European Association for Machine Translation 2017 Convolutional over Recurrent Encoder for Neural Machine Translation Praveen Dakwale and Christof Monz Neural Machine Translation End to end neural network with RNN

354 views • 20 slides
Machine Translation 12: (Non-neural) Statistical Machine Translation Rico Sennrich University of

Machine Translation 12: (Non-neural) Statistical Machine Translation Rico Sennrich University of

Machine Translation 12: (Non-neural) Statistical Machine Translation Rico Sennrich University of Edinburgh R. Sennrich MT 2018 12 1 / 27 Todays Lecture So far, main focus of lecture was on: neural machine translation research

740 views • 46 slides
Adaptive Multi-pass Decoder for Neural Machine Translation EMNLP 2018

Adaptive Multi-pass Decoder for Neural Machine Translation EMNLP 2018

Adaptive Multi-pass Decoder for Neural Machine Translation EMNLP 2018 http://aclweb.org/anthology/D18-1048 Neural Machine Translation(NMT) The encoder-decoder are widely used in neural machine translation the encoder transforms the source

121 views • 11 slides
Semi-supervised Learning for Neural Machine Translation Yong Cheng joint work with Wei Xu,

Semi-supervised Learning for Neural Machine Translation Yong Cheng joint work with Wei Xu,

Semi-supervised Learning for Neural Machine Translation Yong Cheng joint work with Wei Xu, Zhongjun He, Wei He, Hua Wu, Maosong Sun, Yang Liu 1 Machine Translation Automated translation using computer software 2 Machine Translation Rule-based

650 views • 26 slides
Neural Machine Translation: Breaking the Performance Plateau Rico Sennrich Institute for

Neural Machine Translation: Breaking the Performance Plateau Rico Sennrich Institute for

Neural Machine Translation: Breaking the Performance Plateau Rico Sennrich Institute for Language, Cognition and Computation University of Edinburgh July 4 2016 Rico Sennrich Neural Machine Translation 1 / 15 Is Machine Translation Getting

452 views • 21 slides
What can Statistical Machine Translation teach Neural Machine Translation about Structured

What can Statistical Machine Translation teach Neural Machine Translation about Structured

What can Statistical Machine Translation teach Neural Machine Translation about Structured Prediction? Graham Neubig @ ICLR Workshop on Deep Reinforcement Learning Meets Structured Prediction 5/6/2019 Types of Prediction Types of

1.73k views • 127 slides
Googles Multilingual Neural Machine Translation System: Enabling Zero-Shot Translation

Googles Multilingual Neural Machine Translation System: Enabling Zero-Shot Translation

Googles Multilingual Neural Machine Translation System: Enabling Zero-Shot Translation Author: Melvin Johnson , Mike Schuster , Quoc V. Le, Maxim Krikun, Yonghui Wu, Zhifeng Chen, Nikhil Thorat, Fernanda Vigas, Martin Wattenberg, Greg

344 views • 16 slides
Introduction to Neural Machine Translation Gongbo Tang 16 September 2019 Outline Why Neural

Introduction to Neural Machine Translation Gongbo Tang 16 September 2019 Outline Why Neural

Introduction to Neural Machine Translation Gongbo Tang 16 September 2019 Outline Why Neural Machine Translation ? 1 Introduction to Neural Networks 2 Neural Language Models 3 Gongbo Tang Introduction to Neural Machine Translation 2/38

686 views • 50 slides
BREAKTHROUGHS IN NEURAL MACHINE TRANSLATION Olof Mogren Chalmers University of Technology

BREAKTHROUGHS IN NEURAL MACHINE TRANSLATION Olof Mogren Chalmers University of Technology

BREAKTHROUGHS IN NEURAL MACHINE TRANSLATION Olof Mogren Chalmers University of Technology 2016-09-29 COMING SEMINARS T oday: Olof Mogren Neural Machine Translation October 6: John Wiedenhoeft Fast Bayesian inference in Hidden Markov

607 views • 22 slides
Neural Machine Translation: directions for improvement CMSC 470 Marine Carpuat How can we

Neural Machine Translation: directions for improvement CMSC 470 Marine Carpuat How can we

Neural Machine Translation: directions for improvement CMSC 470 Marine Carpuat How can we improve on state-of-the-art machine translation approaches? Model Training Data Objective Algorithm Addressing domain mismatch

490 views • 36 slides
Sparse and Constrained Attention for Neural Machine Translation Chaitanya Malaviya 1 ,

Sparse and Constrained Attention for Neural Machine Translation Chaitanya Malaviya 1 ,

Sparse and Constrained Attention for Neural Machine Translation Chaitanya Malaviya 1 , Pedro Ferreira 2 , Andr F.T. Martins 2,3 1 Carnegie Mellon University, 2 Instituto Superior Tcnico, 3 Unbabel 1 Adequacy in Neural Machine

574 views • 22 slides
An Introduction to Neural Machine Translation Prof. John D. Kelleher @johndkelleher ADAPT Centre

An Introduction to Neural Machine Translation Prof. John D. Kelleher @johndkelleher ADAPT Centre

An Introduction to Neural Machine Translation Prof. John D. Kelleher @johndkelleher ADAPT Centre for Digital Content Technology Dublin Institute of Technology, Ireland June 25, 2018 1 / 57 Outline The Neural Machine Translation Revolution

610 views • 57 slides
Neural machine translation with less supervision CMSC 470 Marine Carpuat Neural MT only helps

Neural machine translation with less supervision CMSC 470 Marine Carpuat Neural MT only helps

Neural machine translation with less supervision CMSC 470 Marine Carpuat Neural MT only helps in high-resource settings Ongoing research Learn from other sources of supervision than pairs (E,F) Monolingual text Multiple languages

479 views • 29 slides
Neural Machine Translation Decoding Philipp Koehn 8 October 2020 Philipp Koehn Machine

Neural Machine Translation Decoding Philipp Koehn 8 October 2020 Philipp Koehn Machine

Neural Machine Translation Decoding Philipp Koehn 8 October 2020 Philipp Koehn Machine Translation: Neural Machine Translation Decoding 8 October 2020 Inference 1 Given a trained model ... we now want to translate test sentences We

682 views • 39 slides
Syntactic Translation Lattices Felix Stahlberg, Adria de Gispert, Eva Hasler, and Bill Byrne

Syntactic Translation Lattices Felix Stahlberg, Adria de Gispert, Eva Hasler, and Bill Byrne

Neural Machine Translation by Minimising the Bayes-risk with Respect to Syntactic Translation Lattices Felix Stahlberg, Adria de Gispert, Eva Hasler, and Bill Byrne Department of Engineering Neural Machine Translation by Minimising the

1.97k views • 12 slides
The Local Amsterdam Cultural Heritage Linked Open Data Network Lukas Koster ( Library of the

The Local Amsterdam Cultural Heritage Linked Open Data Network Lukas Koster ( Library of the

The Local Amsterdam Cultural Heritage Linked Open Data Network Lukas Koster ( Library of the University of Amsterdam) Ivo Zandhuis ( Ivo Zandhuis Research &amp; Consultancy ) SWIB 2018 Bonn AdamNet Foundation: Amsterdam based Library

311 views • 27 slides
Machine Learning: Der KDD-Prozess Knowledge Discovery in Databases Machine Learning Data-Mining

Machine Learning: Der KDD-Prozess Knowledge Discovery in Databases Machine Learning Data-Mining

Machine Learning: Der KDD-Prozess Knowledge Discovery in Databases Machine Learning Data-Mining Aufgaben Aufgabe Von Daten zum Wissen Wissensextraktion durch We are drowing in automatisches Erkennen von information,

1.01k views • 47 slides
Customer Centric Software Project Management Tomas Nystrm 21.4.2005 Accenture Company

Customer Centric Software Project Management Tomas Nystrm 21.4.2005 Accenture Company

Customer Centric Software Project Management Tomas Nystrm 21.4.2005 Accenture Company Background Global Over 100 000 people In Finland we are 650+ Accenture Traditional consulting Accenture Services

886 views • 26 slides
Unsupervised Morpheme Analysis Competition 3: Statistical Machine Translation Mikko Kurimo, Sami

Unsupervised Morpheme Analysis Competition 3: Statistical Machine Translation Mikko Kurimo, Sami

Unsupervised Morpheme Analysis Competition 3: Statistical Machine Translation Mikko Kurimo, Sami Virpioja, Ville T. Turunen (TKK) Graeme W. Blackwood, William Byrne (UCAM) Morphology and SMT Statistical machine translation systems find

333 views • 21 slides
1 min 1-1 P ( W ) , W = w ; w ; : : : ; w 1 2 n Basic Language Modeling Estimate

1 min 1-1 P ( W ) , W = w ; w ; : : : ; w 1 2 n Basic Language Modeling Estimate

Exploiting Syntactic Structure for Language Modeling Basic Language Modeling: Ciprian Chelba, Frederick Jelinek Model Component Parameterization A Structured Language Model: Pruning Strategy Language Model Requirements Word

579 views • 31 slides
Exploiting Syntactic Structure for Language Modeling Ciprian Chelba, Frederick Jelinek

Exploiting Syntactic Structure for Language Modeling Ciprian Chelba, Frederick Jelinek

Hierarchical Structure in Natural Language ended_S ended_VP with_PP loss_NP of_PP cents_NP contract_NP loss_NP cents_NNS 7_CD the_DT contract_NN ended_VBD with_IN a_DT loss_NN of_IN Words are hierarchically organized in

768 views • 39 slides
EAPCI 2018 Expert Consensus Document on Clinical Use of Intracoronary Imaging Giulio Guagliumi,

EAPCI 2018 Expert Consensus Document on Clinical Use of Intracoronary Imaging Giulio Guagliumi,

EAPCI 2018 Expert Consensus Document on Clinical Use of Intracoronary Imaging Giulio Guagliumi, MD FESC EAPC PCI I Sci cientifi fic Docu cument Co Committee Robert Byrne, MD Davide Capodanno, MD Motivation The clinical use of

597 views • 22 slides

More recommend