representing huge translation models statistical machine
play

Representing Huge Translation Models Statistical Machine - PowerPoint PPT Presentation

Jul 05, 2023 •162 likes •1.27k views

Representing Huge Translation Models Statistical Machine Translation parallel text + alignment Statistical Machine Translation extract rules parallel text + alignment Statistical Machine Translation score extract rules rules parallel

  1. Suffix Arrays it makes him and it mars him . it sets him on and it takes him off . # 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 Text T 3 12 2 15 10 6 0 4 8 13 1 5 16 11 9 14 7 17 18 Suffix Array SA O ( | w | log | T | ) (Manber & Myers, 93) O ( | w | + log | T | ) him and it O ( | w | ) (Abouelhoda et al., 04) Query Pattern w

  2. Suffix Arrays it makes him and it mars him . it sets him on and it takes him off . # 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 Text T 3 12 2 15 10 6 0 4 8 13 1 5 16 11 9 14 7 17 18 Suffix Array SA O ( | w | log | T | ) (Manber & Myers, 93) O ( | w | + log | T | ) him and it O ( | w | ) (Abouelhoda et al., 04) on baseline model: Query Pattern w 0.009 seconds/sentence (not including extraction/scoring)

  3. Problem: Phrases with Gaps • Hierarchical phrase-based translation (Chiang 2005, 2007) • Quirk et al. 2005, Simard et al. 2005, DeNeefe et al. 2007 Input it persuades him and it disheartens him Source Phrase it X him

  4. Hierarchical Phrases: Phrases with Gaps • Hierarchical phrase-based translation (Chiang 2005, 2007) • Quirk et al. 2005, Simard et al. 2005, DeNeefe et al. 2007 Input it persuades him and it disheartens him Source Phrase it X him

  5. Hierarchical Phrases: Phrases with Gaps • Hierarchical phrase-based translation (Chiang 2005, 2007) • Quirk et al. 2005, Simard et al. 2005, DeNeefe et al. 2007 Input it persuades him and it disheartens him Source Phrase it X him

  6. Hierarchical Phrases: Phrases with Gaps • Hierarchical phrase-based translation (Chiang 2005, 2007) • Quirk et al. 2005, Simard et al. 2005, DeNeefe et al. 2007 Input it persuades him and it disheartens him Source Phrase it X him

  7. Hierarchical Phrases: Phrases with Gaps • Hierarchical phrase-based translation (Chiang 2005, 2007) • Quirk et al. 2005, Simard et al. 2005, DeNeefe et al. 2007 Input it persuades him and it disheartens him Source Phrase it X and X him

  8. Problem Statement Given an input sentence, efficiently find all hierarchical phrase-based translation rules for that sentence in the training corpus.

  9. Pattern Matching for Hierachical PBMT Input Pattern it persuades him and it disheartens him

  10. Pattern Matching for Hierarchical PBMT Input Pattern it persuades him and it disheartens him it persuades him it Query Patterns persuades him and persuades him and it him and it disheartens and it disheartens him disheartens it persuades him and it persuades persuades him and it persuades him him and it disheartens him and and it disheartens him and it it persuades him and it it disheartens persuades him and it disheartens disheartens him him and it disheartens him

  11. Pattern Matching for Hierarchical PBMT Input Pattern it persuades him and it disheartens him it X and it X disheartens him Query Patterns it X it it X and X him it X disheartens persuades him X disheartens it X him persuades him X him persuades X it persuades X it disheartens persuades X disheartens persuades X disheartens him persuades X him him and X him it persuades X it him X disheartens him it persuades X disheartens it persuades him X disheartens it persuades X him it persuades him X him it X and it it persuades X it disheartens it X it disheartens it persuades X disheartens him

  12. Pattern Matching for Hierarchical PBMT Input Pattern it persuades him and it disheartens him it X and it disheartens Query Patterns it X it disheartens him persuades him and X him persuades him X disheartens him persuades X it disheartens him it persuades him and X him it persuades him X disheartens him it persuades X it disheartens him it X and it disheartens him

  13. Pattern Matching for Hierarchical PBMT Input Pattern it persuades him and it disheartens him it X and it disheartens Query Patterns it X it disheartens him persuades him and X him persuades him X disheartens him persuades X it disheartens him it persuades him and X him it persuades him X disheartens him it persuades X it disheartens him it X and it disheartens him This is a variant of approximate pattern matching (Navarro ‘01)

  14. Pattern Matching with Gaps 3 and it mars him , it sets him ... Query pattern 12 and it takes him off . # α 2 him and it mars him . it sets ... him X it him off . # 15 him on and it takes him off . # 10 him , it sets him on and it ... 6 it makes him and it mars ... 0 it mars him , it sets him on ... 4 it sets him on and it takes ... 8 it takes him off . # 13 makes him and it mars him ... 1 ...

  15. Pattern Matching with Gaps 3 and it mars him , it sets him ... Query pattern α 12 and it takes him off . # 2 him and it mars him . it sets ... him X it him off . # 15 him on and it takes him off . # 10 him , it sets him on and it ... 6 it makes him and it mars ... 0 it mars him , it sets him on ... 4 it sets him on and it takes ... 8 it takes him off . # 13 makes him and it mars him ... 1 ...

  16. Pattern Matching with Gaps 3 and it mars him , it sets him ... Query pattern α 12 and it takes him off . # 2 him and it mars him . it sets ... him X it him off . # 15 him on and it takes him off . # 10 him , it sets him on and it ... 6 it makes him and it mars ... 0 it mars him , it sets him on ... 4 it sets him on and it takes ... 8 it takes him off . # 13 makes him and it mars him ... 1 ...

  17. Pattern Matching with Gaps 3 and it mars him , it sets him ... Query pattern 12 and it takes him off . # α 2 him and it mars him . it sets ... him X it him off . # 15 Subpatterns w i him on and it takes him off . # 10 him , it sets him on and it ... 6 him it makes him and it mars ... 0 it it mars him , it sets him on ... 4 it sets him on and it takes ... 8 it takes him off . # 13 makes him and it mars him ... 1 ...

  18. Pattern Matching with Gaps 3 and it mars him , it sets him ... Query pattern 12 and it takes him off . # α 2 him and it mars him . it sets ... him X it him off . # 15 Subpatterns w i him on and it takes him off . # 10 him , it sets him on and it ... 6 him it makes him and it mars ... 0 it it mars him , it sets him on ... 4 it sets him on and it takes ... 8 it takes him off . # 13 makes him and it mars him ... 1 ...

  19. Pattern Matching with Gaps 3 and it mars him , it sets him ... Query pattern 12 and it takes him off . # α 2 him and it mars him . it sets ... him X it him off . # 15 Subpatterns w i him on and it takes him off . # 10 him , it sets him on and it ... 6 him n i Occurrences it makes him and it mars ... 0 it it mars him , it sets him on ... 4 it sets him on and it takes ... 8 it takes him off . # 13 makes him and it mars him ... 1 ...

  20. Pattern Matching with Gaps 3 and it mars him , it sets him ... 12 and it takes him off . # 2 him and it mars him . it sets ... him off . # 15 2 0 him on and it takes him off . # 10 15 4 him , it sets him on and it ... 6 10 8 it makes him and it mars ... 0 6 13 it mars him , it sets him on ... 4 it sets him on and it takes ... 8 it takes him off . # 13 makes him and it mars him ... 1 ...

  21. Pattern Matching with Gaps 2 0 15 4 10 8 6 13

  22. Pattern Matching with Gaps (2, 4) 2 0 (2, 8) 15 4 (2, 13) 10 8 (6, 8) 6 13 (6, 13) (10, 13)

  23. Pattern Matching with Gaps (2, 4) 2 0 (2, 8) 15 4 (2, 13) 10 8 (6, 8) 6 13 (6, 13) (10, 13) RILMS (Rahman et al., 06)

  24. Pattern Matching with Gaps (2, 4) 2 0 (2, 8) 15 4 (2, 13) 10 8 (6, 8) 6 13 (6, 13) (10, 13) RILMS (Rahman et al., 06) � linear in number of occurrences of subpatterns: O ( n i ) i

  25. Baseline Timing Result 221 seconds per sentence compare: 0.009 seconds per sentence for contiguous phrases

  26. Complexity Analysis � ( | w | + log | T | ) contiguous w 137 5 27 I � � discontiguous ( | w i | + log | T | + n i ) α = w 1 X...Xw I i =1 2825 3 5 27 82069

  27. Complexity Analysis � ( | w | + log | T | ) contiguous w 137 5 27 I � � discontiguous ( | w i | + log | T | + n i ) α = w 1 X...Xw I i =1 2825 3 5 27 82069

  28. Exploiting Redundancy Input Pattern it persuades him and it disheartens him it X and it X disheartens him Query Patterns it X it it X and X him it X disheartens persuades him X disheartens it X him persuades him X him persuades X it persuades X it disheartens persuades X disheartens persuades X disheartens him persuades X him him and X him it persuades X it him X disheartens him it persuades X disheartens it persuades him X disheartens it persuades X him it persuades him X him it X and it it persuades X it disheartens it X it disheartens it persuades X disheartens him

  29. Exploiting Redundancy Input Pattern it persuades him and it disheartens him it X and it X disheartens him Query Patterns it X it it X and X him it X disheartens persuades him X disheartens it X him persuades him X him persuades X it persuades X it disheartens persuades X disheartens persuades X disheartens him persuades X him him and X him it persuades X it him X disheartens him it persuades X disheartens it persuades him X disheartens it persuades X him it persuades him X him it X and it it persuades X it disheartens it X it disheartens it persuades X disheartens him

  30. Exploiting Redundancy Query Pattern it persuades X disheartens him

  31. Exploiting Redundancy Query Pattern it persuades X disheartens him Maximal Prefix it persuades X disheartens (Zhang & Vogel 2005)

  32. Exploiting Redundancy Query Pattern it persuades X disheartens him Maximal Prefix it persuades X disheartens Maximal Suffix persuades X disheartens him

  33. Prefix Tree with Suffix Links him persuades him persuades it X him X him him

  34. Timing Results 221 seconds/ sentence Baseline

  35. Timing Results 221 177 seconds/ sentence Baseline Prefix Tree

  36. Complexity Analysis � ( | w | + log | T | ) contiguous w 137 5 27 I � � discontiguous ( | w i | + log | T | + n i ) α = w 1 X...Xw I i =1 2825 3 5 27 82069

  37. Complexity Analysis � ( | w | + log | T | ) contiguous w 137 5 27 I � � discontiguous ( | w i | + log | T | + n i ) α = w 1 X...Xw I i =1 2825 3 5 27 82069

  38. Empirical Analysis cumulative time (s) computations (ranked by time)

  39. Distribution of Patterns in Training Data Frequency Pattern types (in descending order of frequency)

  40. Distribution of Patterns in Training Data Frequency Pattern types (in descending order of frequency)

  41. Analysis of Problem • The expensive computations involve at least one frequent subpattern. There are two cases. • A frequent pattern paired with an infrequent pattern • Two frequent patterns paired with each other

  42. Frequent × Infrequent Subpatterns

  43. Frequent × Infrequent Subpatterns

  44. Frequent × Infrequent Subpatterns

  45. Frequent × Infrequent Subpatterns

  46. Double Binary Search Baeza-Yates, 04

  47. Double Binary Search Baeza-Yates, 04 Queryset Q Dataset D

  48. Double Binary Search Baeza-Yates, 04 Queryset Q Dataset D

  49. Double Binary Search Baeza-Yates, 04 Queryset Q Dataset D

  50. Double Binary Search Baeza-Yates, 04 Queryset Q Dataset D

  51. Double Binary Search Baeza-Yates, 04 Queryset Q Dataset D

  52. Double Binary Search Baeza-Yates, 04 Queryset Q Dataset D

  53. Double Binary Search Baeza-Yates, 04 Queryset Q Dataset D complexity: | Q | log | D | Upper bound

  54. Obtaining Sorted Sets

  55. Obtaining Sorted Sets Sort via Stratified Tree (van Emde Boas et al. 1977)

  56. Obtaining Sorted Sets Sort via Stratified Tree (van Emde Boas et al. 1977) Problem: complexity increases to O ( | Q | log | D | + ( | Q | + | D | ) log log | T | )

  57. Obtaining Sorted Sets Sort via Stratified Tree (van Emde Boas et al. 1977) Solution: cache sorted set in prefix tree Problem: complexity increases to O ( | Q | log | D | + ( | Q | + | D | ) log log | T | )

  58. Timing Results 221 177 seconds/ sentence Baseline Prefix + double Tree binary

  59. Timing Results 221 177 174 seconds/ sentence Baseline Prefix + double Tree binary

  60. Obtaining Sorted Sets Sort via Stratified Tree Problem: sort complexity is still very high for very frequent patterns

  61. Obtaining Sorted Sets Solution: precompute the inverted index for 1000 most frequent contiguous patterns

  62. Timing Results 221 177 174 seconds/ sentence Baseline Prefix + double Tree binary

  63. Timing Results 221 177 174 seconds/ sentence 44 Baseline Prefix + double + inverted Tree binary indices

  64. Frequent × Frequent Subpatterns

  65. Frequent × Frequent Subpatterns Problem: There is no clever algorithm to solve this problem

  66. Solution: Precomputation it makes him and it mars him . it sets him on and it takes him off . # it makes him and it mars him . it sets him on and it takes him off . # 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 Text

Download Presentation
Download Policy: The content available on the website is offered to you 'AS IS' for your personal information and use only. It cannot be commercialized, licensed, or distributed on other websites without prior consent from the author. To download a presentation, simply click this link. If you encounter any difficulties during the download process, it's possible that the publisher has removed the file from their server.

Recommend

Chapter 11 Tree-based models Statistical Machine Translation Tree-Based Models Traditional

Chapter 11 Tree-based models Statistical Machine Translation Tree-Based Models Traditional

Chapter 11 Tree-based models Statistical Machine Translation Tree-Based Models Traditional statistical models operate on sequences of words Many translation problems can be best explained by pointing to syntax reordering, e.g., verb

1.39k views • 120 slides
Chapter 4 Word-based models Statistical Machine Translation Lexical Translation How to

Chapter 4 Word-based models Statistical Machine Translation Lexical Translation How to

Chapter 4 Word-based models Statistical Machine Translation Lexical Translation How to translate a word look up in dictionary Haus house, building, home, household, shell. Multiple translations some more frequent than others

1.2k views • 61 slides
Statistical Machine Translation George Foster George Foster Statistical Machine Translation A

Statistical Machine Translation George Foster George Foster Statistical Machine Translation A

Statistical Machine Translation George Foster George Foster Statistical Machine Translation A Brief History of MT Origins (1949): WW II codebreaking success suggests statistical approach to MT George Foster Statistical Machine Translation A

840 views • 54 slides
Statistical Machine Translation Nadir Durrani 21-November-2014 Machine Translation

Statistical Machine Translation Nadir Durrani 21-November-2014 Machine Translation

Statistical Machine Translation Nadir Durrani 21-November-2014 Machine Translation www.uni-stuttart.de Problem: Automatic translation the foreign text: 2 Open Problems in Machine Translation www.uni-stuttart.de Ambiguity in translation

938 views • 44 slides
Chapter 7 Language models Statistical Machine Translation Language models Language models

Chapter 7 Language models Statistical Machine Translation Language models Language models

Chapter 7 Language models Statistical Machine Translation Language models Language models answer the question: How likely is a string of English words good English? Help with reordering p lm (the house is small) > p lm (small the is

1.06k views • 51 slides
Compositions of Tree-to-Tree Statistical Machine Translation Models Andreas Maletti Institute

Compositions of Tree-to-Tree Statistical Machine Translation Models Andreas Maletti Institute

Compositions of Tree-to-Tree Statistical Machine Translation Models Andreas Maletti Institute for Natural Language Processing University of Stuttgart, Germany Montral July 28, 2016 Compositions of Tree-to-Tree Statistical Machine

290 views • 25 slides
Statistical Machine Translation Graham Neubig Nara Institute of Science and Technology (NAIST)

Statistical Machine Translation Graham Neubig Nara Institute of Science and Technology (NAIST)

Statistical Machine Translation Statistical Machine Translation Graham Neubig Nara Institute of Science and Technology (NAIST) 10/23/2012 1 Statistical Machine Translation Machine Translation Automatically translate between languages

792 views • 42 slides
Improved Models of Distortion Cost for Statistical Machine Translation Spence Green, Michel

Improved Models of Distortion Cost for Statistical Machine Translation Spence Green, Michel

Improved Models of Distortion Cost for Statistical Machine Translation Spence Green, Michel Galley, and Christopher D. Manning Stanford University June 4, 2010 Motivation A New Cost Model Phrase-based MT Evaluation Limit v. Cost Conclusion

520 views • 39 slides
Chapter 5 Phrase-based models Statistical Machine Translation Motivation Word-Based Models

Chapter 5 Phrase-based models Statistical Machine Translation Motivation Word-Based Models

Chapter 5 Phrase-based models Statistical Machine Translation Motivation Word-Based Models translate words as atomic units Phrase-Based Models translate phrases as atomic units Advantages: many-to-many translation can handle

303 views • 27 slides
Meta-Learning for Low Resource NMT Introduction Historically Statistical Translation

Meta-Learning for Low Resource NMT Introduction Historically Statistical Translation

Meta-Learning for Low Resource NMT Introduction Historically Statistical Translation Neural Machine Translation recently outperforms Statistical Models outperformed translations on low resource language pairs NMT Previous Work

426 views • 19 slides
Statistical Machine Translation Statistical Machine Translation p Lecture 2 Theory and Praxis of

Statistical Machine Translation Statistical Machine Translation p Lecture 2 Theory and Praxis of

Components: Translation model, language model, decoder Statistical Machine Translation Lecture 2: Theory and Praxis of Decoding p Statistical Machine Translation Statistical Machine Translation p Lecture 2 Theory and Praxis of Decoding

540 views • 9 slides
Statistical Machine Translation Outline p Why Syntax? Lecture 5 Yamada and Knight:

Statistical Machine Translation Outline p Why Syntax? Lecture 5 Yamada and Knight:

Reminder: Modeling and Decoding Syntax-Based Statistical Machine Translationp Statistical Machine Translation Outline p Why Syntax? Lecture 5 Yamada and Knight: translating into trees Syntax-Based Models Wu: tree-based transfer

106 views • 7 slides
Machine Translation 12: (Non-neural) Statistical Machine Translation Rico Sennrich University of

Machine Translation 12: (Non-neural) Statistical Machine Translation Rico Sennrich University of

Machine Translation 12: (Non-neural) Statistical Machine Translation Rico Sennrich University of Edinburgh R. Sennrich MT 2018 12 1 / 27 Todays Lecture So far, main focus of lecture was on: neural machine translation research

740 views • 46 slides
What can Statistical Machine Translation teach Neural Machine Translation about Structured

What can Statistical Machine Translation teach Neural Machine Translation about Structured

What can Statistical Machine Translation teach Neural Machine Translation about Structured Prediction? Graham Neubig @ ICLR Workshop on Deep Reinforcement Learning Meets Structured Prediction 5/6/2019 Types of Prediction Types of

1.73k views • 127 slides
Chapter 8 Evaluation Statistical Machine Translation Evaluation How good is a given machine

Chapter 8 Evaluation Statistical Machine Translation Evaluation How good is a given machine

Chapter 8 Evaluation Statistical Machine Translation Evaluation How good is a given machine translation system? Hard problem, since many different translations acceptable semantic equivalence / similarity Evaluation metrics

456 views • 42 slides
3 Language Models 1: n -gram Language Models While the final goal of a statistical machine

3 Language Models 1: n -gram Language Models While the final goal of a statistical machine

3 Language Models 1: n -gram Language Models While the final goal of a statistical machine translation system is to create a model of the target sentence E given the source sentence F , P ( E | F ), in this chapter we will take a step back, and

106 views • 8 slides
Refining Network Intents for Self-Driving Networks Arthur Selle Jacobs Ricardo Jos

Refining Network Intents for Self-Driving Networks Arthur Selle Jacobs Ricardo Jos

Refining Network Intents for Self-Driving Networks Arthur Selle Jacobs Ricardo Jos Pfitscher, Ronaldo Alves Ferreira, Lisandro Zambenedetti Granville UFRGS UFMS Budapest, Hungary August 24, 2018 Self-Driving Networks High-level

874 views • 31 slides
Oklo: case study in extracting information on fundamental interactions from CN processes Edward

Oklo: case study in extracting information on fundamental interactions from CN processes Edward

Oklo: case study in extracting information on fundamental interactions from CN processes Edward Davis edward.davis@ku.edu.kw Kuwait University ACFI Workshop: Tests of Time-Reversal in Nuclear and Hadronic Processes (November 6 to 8,

691 views • 45 slides
A Control Point for Reducing Root Abuse of File-System Privileges Glenn Wurster , Paul C. van

A Control Point for Reducing Root Abuse of File-System Privileges Glenn Wurster , Paul C. van

Introduction Solution for the Desktop A Control Point for Reducing Root Abuse of File-System Privileges Glenn Wurster , Paul C. van Oorschot School of Computer Science Carleton University, Canada 6 Oct 2010 Glenn Wurster, Paul C. van Oorschot

527 views • 25 slides
New Slide Attacks on Almost Self-Similar Ciphers Orr Dunkelman, Nathan Keller, Noam Lasry, Adi

New Slide Attacks on Almost Self-Similar Ciphers Orr Dunkelman, Nathan Keller, Noam Lasry, Adi

Intro SelfSimilar New Summary New Slide Attacks on Almost Self-Similar Ciphers Orr Dunkelman, Nathan Keller, Noam Lasry, Adi Shamir Eurocrypt 2020 Orr Dunkelman New Slide Attacks on Almost Self-Similar Ciphers 1/ 28 Intro SelfSimilar

819 views • 40 slides
Household data analytics Dagstuhl, February 2015 Christoph Doblander, Anwar Ul Haq, Christoph

Household data analytics Dagstuhl, February 2015 Christoph Doblander, Anwar Ul Haq, Christoph

Technische Universitt Mnchen Household data analytics Dagstuhl, February 2015 Christoph Doblander, Anwar Ul Haq, Christoph Goebel, H.-A. Jacobsen Technische Universitt Mnchen Institut fr Informatik, Lehrstuhl I13 Prof. Dr. H.-A.

462 views • 14 slides
Malicious Code Malicious Code for Fun and Profit for Fun and Profit Mihai Christodorescu

Malicious Code Malicious Code for Fun and Profit for Fun and Profit Mihai Christodorescu

Malicious Code Malicious Code for Fun and Profit for Fun and Profit Mihai Christodorescu mihai@cs.wisc.edu 29 March 2007 SYN Cookies (contd) SYN Cookies (contd) SYN cookies are particular choices of initial TCP sequence numbers

868 views • 62 slides
Malicious Code Malicious Code for Fun and Profit for Fun and Profit Mihai Christodorescu

Malicious Code Malicious Code for Fun and Profit for Fun and Profit Mihai Christodorescu

Malicious Code Malicious Code for Fun and Profit for Fun and Profit Mihai Christodorescu mihai@cs.wisc.edu 23 March 2006 What is Malicious Code? What is Malicious Code? Viruses, worms, trojans, Code that breaks your security policy.

707 views • 69 slides
Unik Idit Levine EMC CONFIDENTIALINTERNAL USE ONLY EMC CONFIDENTIALINTERNAL USE ONLY 1

Unik Idit Levine EMC CONFIDENTIALINTERNAL USE ONLY EMC CONFIDENTIALINTERNAL USE ONLY 1

Unik Idit Levine EMC CONFIDENTIALINTERNAL USE ONLY EMC CONFIDENTIALINTERNAL USE ONLY 1 Virtualization Stack Application Config Application The aim is to run single Language Runtime Application with a single user on a single server

2.88k views • 28 slides

More recommend