5th quality estimation shared task
play

5th Quality Estimation Shared Task WMT16 Lucia Specia, Varvara - PowerPoint PPT Presentation

Nov 09, 2022 •703 likes •1.03k views

Overview T1-Sentence-level HTER T2-Word-level OK/BAD T2p-Phrase-level OK/BAD T3-Document-level PE Discussion 5th Quality Estimation Shared Task WMT16 Lucia Specia, Varvara Logacheva and Carolina Scarton University of Sheffield Berlin, 12

  1. Overview T1-Sentence-level HTER T2-Word-level OK/BAD T2p-Phrase-level OK/BAD T3-Document-level PE Discussion 5th Quality Estimation Shared Task WMT16 Lucia Specia, Varvara Logacheva and Carolina Scarton University of Sheffield Berlin, 12 August 2016 5th Quality Estimation Shared Task 1 / 25

  2. Overview T1-Sentence-level HTER T2-Word-level OK/BAD T2p-Phrase-level OK/BAD T3-Document-level PE Discussion Outline Overview 1 T1-Sentence-level HTER 2 T2-Word-level OK/BAD 3 T2p-Phrase-level OK/BAD 4 T3-Document-level PE 5 Discussion 6 5th Quality Estimation Shared Task 2 / 25

  3. Overview T1-Sentence-level HTER T2-Word-level OK/BAD T2p-Phrase-level OK/BAD T3-Document-level PE Discussion Goals QE metrics predict the quality of a translated text without a reference translation Goals in 2016 : Advance work on sentence and word-level QE High quality datasets, professionally post-edited Introduce a phrase-level task Introduce a document-level task 5th Quality Estimation Shared Task 3 / 25

  4. Overview T1-Sentence-level HTER T2-Word-level OK/BAD T2p-Phrase-level OK/BAD T3-Document-level PE Discussion Tasks T1: Predicting sentence-level post-editing (PE) distance T2: Predicting word and phrase-level OK/BAD labels T3: Predicting document-level 2-stage PE distance 5th Quality Estimation Shared Task 4 / 25

  5. Overview T1-Sentence-level HTER T2-Word-level OK/BAD T2p-Phrase-level OK/BAD T3-Document-level PE Discussion Participants ID Team CDACM Centre for Development of Advanced Computing, India POSTECH Pohang University of Science and Technology, Republic of Korea RTM Referential Translation Machines, Turkey SHEF University of Sheffield, UK SHEF-LIUM University of Sheffield, UK and Laboratoire d’Informatique de l’Universit´ e du Maine, France SHEF-MIME University of Sheffield, UK UAlacant University of Alicante, Spain UFAL Nile University, Egypt & Charles University, Czech Republic UGENT Ghent University, Belgium UNBABEL Unbabel, Portugal USFD University of Sheffield, UK USHEF University of Sheffield, UK UU Uppsala University, Sweden YSDA Yandex, Russia 14 teams, 39 systems : up to 2 per team, per subtask 5th Quality Estimation Shared Task 5 / 25

  6. Overview T1-Sentence-level HTER T2-Word-level OK/BAD T2p-Phrase-level OK/BAD T3-Document-level PE Discussion Outline Overview 1 T1-Sentence-level HTER 2 T2-Word-level OK/BAD 3 T2p-Phrase-level OK/BAD 4 T3-Document-level PE 5 Discussion 6 5th Quality Estimation Shared Task 6 / 25

  7. Overview T1-Sentence-level HTER T2-Word-level OK/BAD T2p-Phrase-level OK/BAD T3-Document-level PE Discussion Predicting sentence-level HTER Languages, data and MT systems 12K/1K/2K train/dev/test English → German ( QT21 ) One SMT system IT domain Post-edited by professional translators Labelling: HTER Instances: < SRC, MT, PE, HTER > 5th Quality Estimation Shared Task 7 / 25

  8. Overview T1-Sentence-level HTER T2-Word-level OK/BAD T2p-Phrase-level OK/BAD T3-Document-level PE Discussion Predicting sentence-level HTER System ID Pearson ↑ Spearman ↑ English-German • YSDA/SNTX+BLEU+SVM 0.525 – POSTECH/SENT-RNN-QV2 0.460 0.483 SHEF-LIUM/SVM-NN-emb-QuEst 0.451 0.474 POSTECH/SENT-RNN-QV3 0.447 0.466 SHEF-LIUM/SVM-NN-both-emb 0.430 0.452 UGENT-LT3/SCATE-SVM2 0.412 0.418 UFAL/MULTIVEC 0.377 0.410 RTM/RTM-FS-SVR 0.376 0.400 UU/UU-SVM 0.370 0.405 UGENT-LT3/SCATE-SVM1 0.363 0.375 RTM/RTM-SVR 0.358 0.384 Baseline SVM 0.351 0.390 SHEF/SimpleNets-SRC 0.182 – SHEF/SimpleNets-TGT 0.182 – • = winning submissions - top-scoring and those which are not significantly worse. Gray area = systems that are not significantly different from the baseline. 5th Quality Estimation Shared Task 8 / 25

  9. Overview T1-Sentence-level HTER T2-Word-level OK/BAD T2p-Phrase-level OK/BAD T3-Document-level PE Discussion Predicting sentence-level HTER: 2016 vs 2015 Different language pair, different domain, different MT system: System ID ( 2015 ) Pearson’s r ↑ English-Spanish • LORIA/17+LSI+MT+FILTRE 0.39 • LORIA/17+LSI+MT 0.39 • RTM-DCU/RTM-FS+PLS-SVR 0.38 RTM-DCU/RTM-FS-SVR 0.38 UGENT-LT3/SCATE-SVM 0.37 UGENT-LT3/SCATE-SVM-single 0.32 SHEF/SVM 0.29 SHEF/GP 0.19 Baseline SVM 0.14 5th Quality Estimation Shared Task 9 / 25

  10. Overview T1-Sentence-level HTER T2-Word-level OK/BAD T2p-Phrase-level OK/BAD T3-Document-level PE Discussion Outline Overview 1 T1-Sentence-level HTER 2 T2-Word-level OK/BAD 3 T2p-Phrase-level OK/BAD 4 T3-Document-level PE 5 Discussion 6 5th Quality Estimation Shared Task 10 / 25

  11. Overview T1-Sentence-level HTER T2-Word-level OK/BAD T2p-Phrase-level OK/BAD T3-Document-level PE Discussion Predicting word-level quality Languages, data and MT systems Same as for T1 Labelling done with TERCOM: OK = unchanged BAD = insertion, substitution Instances: < source word, MT word, OK/BAD label > Sentences Words % of BAD words Training 12 , 000 210 , 958 21 . 4 Dev 1 , 000 19 , 487 19 . 54 Test 2 , 000 34 , 531 19 . 31 Challenge : skewed class distribution 5th Quality Estimation Shared Task 11 / 25

  12. Overview T1-Sentence-level HTER T2-Word-level OK/BAD T2p-Phrase-level OK/BAD T3-Document-level PE Discussion Predicting word-level quality Mostly interested in finding errors Precision/recall preferences depend on application Rare classes should not dominate New evaluation metric : F 1 -multiplied = F 1 -OK × F 1 -BAD Baseline : CRF classifier with 22 features 5th Quality Estimation Shared Task 12 / 25

  13. Overview T1-Sentence-level HTER T2-Word-level OK/BAD T2p-Phrase-level OK/BAD T3-Document-level PE Discussion Predicting word-level quality System ID F 1 - mult ↑ F 1 -BAD F 1 -OK English-German • UNBABEL/ensemble 0.495 0.560 0.885 UNBABEL/linear 0.463 0.529 0.875 UGENT-LT3/SCATE-RF 0.411 0.492 0.836 UGENT-LT3/SCATE-ENS 0.381 0.464 0.821 POSTECH/WORD-RNN-QV3 0.380 0.447 0.850 POSTECH/WORD-RNN-QV2 0.376 0.454 0.828 UAlacant/SBI-Online-baseline 0.367 0.456 0.805 CDACM/RNN 0.353 0.419 0.842 SHEF/SHEF-MIME-1 0.338 0.403 0.839 SHEF/SHEF-MIME-0.3 0.330 0.391 0.845 Baseline CRF 0.324 0.368 0.880 RTM/s5-RTM-GLMd 0.308 0.349 0.882 UAlacant/SBI-Online 0.290 0.406 0.715 RTM/s4-RTM-GLMd 0.273 0.307 0.888 All OK baseline 0.0 0.0 0.893 All BAD baseline 0.0 0.323 0.0 5th Quality Estimation Shared Task 13 / 25

  14. Overview T1-Sentence-level HTER T2-Word-level OK/BAD T2p-Phrase-level OK/BAD T3-Document-level PE Discussion Predicting word-level quality: 2016 vs 2015 System ID ( 2015 ) F 1 -mult F 1 -OK F 1 -BAD English-Spanish • UAlacant/OnLine-SBI-Baseline 0.336 0.431 0.781 • HDCL/QUETCHPLUS 0.342 0.431 0.794 UAlacant/OnLine-SBI 0.316 0.415 0.761 SAU/KERC-CRF 0.338 0.391 0.864 SAU/KERC-SLG-CRF 0.336 0.389 0.864 SHEF2/W2V-BI-2000 0.275 0.384 0.716 SHEF2/W2V-BI-2000-SIM 0.275 0.384 0.715 SHEF1/QuEst++-AROW 0.259 0.384 0.676 UGENT/SCATE-HYBRID 0.305 0.367 0.830 DCU-SHEFF/BASE-NGRAM-2000 0.273 0.366 0.745 HDCL/QUETCH 0.298 0.353 0.846 DCU-SHEFF/BASE-NGRAM-5000 0.292 0.345 0.845 SHEF1/QuEst++-PA 0.836 0.343 0.244 All BAD baseline 0.00 0.318 0.00 UGENT/SCATE-MBL 0.258 0.306 0.843 RTM-DCU/s5-RTM-GLMd 0.211 0.239 0.881 RTM-DCU/s4-RTM-GLMd 0.200 0.227 0.883 Baseline CRF 0.147 0.168 0.889 All OK baseline 0.00 0.00 0.896 5th Quality Estimation Shared Task 14 / 25

  15. Overview T1-Sentence-level HTER T2-Word-level OK/BAD T2p-Phrase-level OK/BAD T3-Document-level PE Discussion Predicting word-level quality: 2016 vs 2015 Improved baseline New metric: trivial baselines at the bottom Better systems: all submissions outperform all BAD baseline , even in terms of F 1 -BAD 5th Quality Estimation Shared Task 15 / 25

  16. Overview T1-Sentence-level HTER T2-Word-level OK/BAD T2p-Phrase-level OK/BAD T3-Document-level PE Discussion Outline Overview 1 T1-Sentence-level HTER 2 T2-Word-level OK/BAD 3 T2p-Phrase-level OK/BAD 4 T3-Document-level PE 5 Discussion 6 5th Quality Estimation Shared Task 16 / 25

  17. Overview T1-Sentence-level HTER T2-Word-level OK/BAD T2p-Phrase-level OK/BAD T3-Document-level PE Discussion Predicting phrase-level quality Languages, data and MT systems Same as for T1 Labelling: TERCOM + phrase segmentation OK OK OK OK BAD BAD BAD OK Beim Schließen � eines Dokuments � werden � die Historie . OK OK BAD BAD Instances: < source phrase, MT phrase, OK/BAD label > Sentences Phrases % of BAD phrases Training 12 , 000 109 , 921 29 . 84 Dev 1 , 000 9 , 024 30 . 21 Test 2 , 000 16 , 450 29 . 53 5th Quality Estimation Shared Task 17 / 25

  18. Overview T1-Sentence-level HTER T2-Word-level OK/BAD T2p-Phrase-level OK/BAD T3-Document-level PE Discussion Predicting phrase-level quality Languages, data and MT systems Same as for T1 Labelling: TERCOM + phrase segmentation OK OK OK OK BAD BAD BAD OK Beim Schließen � eines Dokuments � werden � die Historie . OK OK BAD BAD Instances: < source phrase, MT phrase, OK/BAD label > Sentences Phrases % of BAD phrases Training 12 , 000 109 , 921 29 . 84 Dev 1 , 000 9 , 024 30 . 21 Test 2 , 000 16 , 450 29 . 53 5th Quality Estimation Shared Task 17 / 25

Download Presentation
Download Policy: The content available on the website is offered to you 'AS IS' for your personal information and use only. It cannot be commercialized, licensed, or distributed on other websites without prior consent from the author. To download a presentation, simply click this link. If you encounter any difficulties during the download process, it's possible that the publisher has removed the file from their server.

Recommend

4th Quality Estimation Shared Task WMT15 Lucia Specia , Chris Hokamp , Varvara Logacheva

4th Quality Estimation Shared Task WMT15 Lucia Specia , Chris Hokamp , Varvara Logacheva

Overview T1 - Sentence-level HTER T2 - Word-level OK/BAD T3 - Paragraph-level Meteor Discussion 4th Quality Estimation Shared Task WMT15 Lucia Specia , Chris Hokamp , Varvara Logacheva and Carolina Scarton University of

517 views • 28 slides
LAW-MWE-CxG 2018 Shared task poster boosters 1. DEEP-BGT AT PARSEME SHARED TASK 2018:

LAW-MWE-CxG 2018 Shared task poster boosters 1. DEEP-BGT AT PARSEME SHARED TASK 2018:

LAW-MWE-CxG 2018 Shared task poster boosters 1. DEEP-BGT AT PARSEME SHARED TASK 2018: BIDIRECTIONAL LSTM- CRF MODEL FOR VERBAL MULTIWORD EXPRESSION IDENTIFICATION, Gzde Berk, Berna Erden and Tunga Gungor 2. GBD-NER AT PARSEME SHARED TASK

555 views • 14 slides
Shared Governance Task Force Report https://web.ramapo.edu/shared-governance-task-force/ 1

Shared Governance Task Force Report https://web.ramapo.edu/shared-governance-task-force/ 1

4/4/2018 Ramapo College of New Jersey Shared Governance Task Force Report https://web.ramapo.edu/shared-governance-task-force/ 1 Overview of Preliminary Report How Did We Get Here? The Charge to the Shared Governance Task Force (SGTF)

228 views • 9 slides
The PARSEME Shared Task on Automatic Identification of Verbal Multiword Expressions (1.1) C.

The PARSEME Shared Task on Automatic Identification of Verbal Multiword Expressions (1.1) C.

Context Annotation methodology Corpora Shared task Wrapping up The PARSEME Shared Task on Automatic Identification of Verbal Multiword Expressions (1.1) C. Ramisch 1 , S. Ricardo 1 , A. Savary 2 , V. Vincze 3 , V. Barbu 4 , A. Bhatia 5 , M.

1.02k views • 25 slides
The SIGMORPHON 2016 shared task morphological reinflection Ryan Cotterell, Christo Kirov,

The SIGMORPHON 2016 shared task morphological reinflection Ryan Cotterell, Christo Kirov,

The SIGMORPHON 2016 shared task morphological reinflection Ryan Cotterell, Christo Kirov, John Sylak-Glassman, David Yarowsky, Jason Eisner, Mans Hulden 1 SIGMORPHON 2016 Shared task - morphological reinflection Shared task

1.28k views • 73 slides
WMT 10 Shared Tasks: Translation Task System Combination Task Chris Callison-Burch, Philipp

WMT 10 Shared Tasks: Translation Task System Combination Task Chris Callison-Burch, Philipp

WMT 10 Shared Tasks: Translation Task System Combination Task Chris Callison-Burch, Philipp Koehn, Christof Monz, Omar Zaidan 15 July 2010 Philipp Koehn WMT10 Shared Tasks 15 July 2010 Translation Task 1 Open benchmark for machine

662 views • 37 slides
7/10/2020 Air Quality Task Force Meeting 7/10/2020 Air Quality Task Force Meeting

7/10/2020 Air Quality Task Force Meeting 7/10/2020 Air Quality Task Force Meeting

7/10/2020 Air Quality Task Force Meeting 7/10/2020 Air Quality Task Force Meeting https://www.portofoakland.com/business/bids rfps/bid engineering/permits/ 7/10/2020 Air Quality Task Force Meeting 7/10/2020 Air Quality Task Force Meeting

439 views • 18 slides
Modeling Unrestricted Coreference in OntoNotes CoNLL-2011 Shared Task Sameer S Pradhan 1 Lance

Modeling Unrestricted Coreference in OntoNotes CoNLL-2011 Shared Task Sameer S Pradhan 1 Lance

CoNLL Shared Task OntoNotes Evaluation Modeling Unrestricted Coreference in OntoNotes CoNLL-2011 Shared Task Sameer S Pradhan 1 Lance Ramshaw 1 Mitchell Marcus 2 Martha Palmer 3 Ralph Weischedel 1 Nianwen Xue 4 1 BBN Technologies, Cambridge, MA

1.17k views • 70 slides
ADAPTIVE QUALITY ESTIMATION FOR MACHINE TRANSLATION AND AUTOMATIC SPEECH RECOGNITION Jos G. C.

ADAPTIVE QUALITY ESTIMATION FOR MACHINE TRANSLATION AND AUTOMATIC SPEECH RECOGNITION Jos G. C.

1 ADAPTIVE QUALITY ESTIMATION FOR MACHINE TRANSLATION AND AUTOMATIC SPEECH RECOGNITION Jos G. C. de Souza Advisors: Matteo Negri Marco Turchi Marcello Federico EAMT 2017 30/05/2017 What is MT Quality Estimation? 2 Translated Source

434 views • 40 slides
Quality Estimation for Language Output Applications Carolina Scarton, Gustavo Paetzold and Lucia

Quality Estimation for Language Output Applications Carolina Scarton, Gustavo Paetzold and Lucia

Quality Estimation for Language Output Applications Carolina Scarton, Gustavo Paetzold and Lucia Specia University of Sheffield, UK COLING, Osaka, 11 Dec 2016 Quality Estimation Approaches to predict the quality of a language output

863 views • 65 slides
Results of the WMT19 Metrics Shared Task Segment-Level and Strong MT Systems Pose Big Challenges

Results of the WMT19 Metrics Shared Task Segment-Level and Strong MT Systems Pose Big Challenges

Results of the WMT19 Metrics Shared Task Segment-Level and Strong MT Systems Pose Big Challenges Qingsong Ma Johnny Tian-Zheng Wei Ond rej Bojar Yvette Graham 1 / 31 Overview Overview of Metrics Task Updates to Metric Task in

554 views • 40 slides
TUPA at MRP 2019 A Multi-Task Baseline System CoNLL Shared Task 3 November 2019 1 / 9 Daniel

TUPA at MRP 2019 A Multi-Task Baseline System CoNLL Shared Task 3 November 2019 1 / 9 Daniel

TUPA at MRP 2019 A Multi-Task Baseline System CoNLL Shared Task 3 November 2019 1 / 9 Daniel Hershcovich 1 , Ofjr Arviv 2 1 University of Copenhagen 2 Hebrew University of Jerusalem Transition-Based UCCA Parser look (Hershcovich et al., 2017)

405 views • 12 slides
Bootstrapping Quality Estimation in a live production environment EAMT 2017 Introduction

Bootstrapping Quality Estimation in a live production environment EAMT 2017 Introduction

Bootstrapping Quality Estimation in a live production environment EAMT 2017 Introduction Quality Estimation The process of scoring Machine Translation (MT) output without access to a reference translation QE aims: Hide bad MT

439 views • 24 slides
Translation Quality Estimation: Past, Present, and Future Andr e Martins MT Marathon, Lisbon,

Translation Quality Estimation: Past, Present, and Future Andr e Martins MT Marathon, Lisbon,

Translation Quality Estimation: Past, Present, and Future Andr e Martins MT Marathon, Lisbon, August 31st, 2017 Andr e Martins (Unbabel) Quality Estimation MTM, 31/8/17 1 / 69 This Talk First part: largely based on Lucia Specias

989 views • 96 slides
HAU at the GermEval 2019 Shared Task on the Identification of Offensive Language in Microposts

HAU at the GermEval 2019 Shared Task on the Identification of Offensive Language in Microposts

HAU at the GermEval 2019 Shared Task on the Identification of Offensive Language in Microposts System Description of Word List, Statistical and Hybrid Approaches afer 1 , Tom De Smedt 2 , and Sylvia Jaki 3 Johannes Sch 1 Institute for

493 views • 28 slides
Sentence-Level Quality Estimation for MT System Combination Tsuyoshi Okita, Rapha el Rubino,

Sentence-Level Quality Estimation for MT System Combination Tsuyoshi Okita, Rapha el Rubino,

Sentence-Level Quality Estimation for MT System Combination Tsuyoshi Okita, Rapha el Rubino, Josef van Genabith Dublin City University Overview Introduction Quality Estimation for System Combination Sentence Level QE Features Extraction

533 views • 20 slides
54 Years of Graph Isomorphism Testing Brendan McKay Australian National University isomorphism

54 Years of Graph Isomorphism Testing Brendan McKay Australian National University isomorphism

54 Years of Graph Isomorphism Testing Brendan McKay Australian National University isomorphism 1 isomorphism 2 The concept of isomorphism We have some objects consisting of some finite sets and some relations on them. An isomorphism

992 views • 72 slides
A Bucket Graph Based Labelling Algorithm for Vehicle Routing Pricing Ruslan Sadykov 1,2 Artur

A Bucket Graph Based Labelling Algorithm for Vehicle Routing Pricing Ruslan Sadykov 1,2 Artur

A Bucket Graph Based Labelling Algorithm for Vehicle Routing Pricing Ruslan Sadykov 1,2 Artur Pessoa 3 Eduardo Uchoa 3 1 2 3 Inria Bordeaux, Universit Bordeaux, Universidade Federal France France Fluminense, Brazil POC Autumn School on

746 views • 60 slides
Diagrams of Affine Permutations and Their Labellings Taedong Yun Oracle June 23, 2014 Taedong

Diagrams of Affine Permutations and Their Labellings Taedong Yun Oracle June 23, 2014 Taedong

Diagrams of Affine Permutations and Their Labellings Taedong Yun Oracle June 23, 2014 Taedong Yun Permutation Diagrams and Labellings June 23, 2014 1 / 36 Contents Based on the joint work with Hwanchul Yoo (KIAS). Taedong Yun

972 views • 55 slides
Adjacency labeling schemes and induced-universal graphs How to save lg n bits Stephen

Adjacency labeling schemes and induced-universal graphs How to save lg n bits Stephen

Adjacency labeling schemes and induced-universal graphs How to save lg n bits Stephen Alstrup Univ. of Copenhagen Haim Kaplan Tel Aviv Univ. Mikkel Thorup Univ. of Copenhagen Uri Zwick Tel Aviv Univ. Adjacency labeling

657 views • 24 slides
Algorithms for CTL B. Srivathsan Chennai Mathematical Institute Model Checking and Systems

Algorithms for CTL B. Srivathsan Chennai Mathematical Institute Model Checking and Systems

Algorithms for CTL B. Srivathsan Chennai Mathematical Institute Model Checking and Systems Verification January - April 2016 1 / 22 Module 1: Adequate CTL formulae 2 / 22 Recap of CTL State formulae := true | p i | 1 2 | 1

1.4k views • 123 slides
SIGML Department of CSE IIT Kanpur Special Interest Group in Machine Learning Classification

SIGML Department of CSE IIT Kanpur Special Interest Group in Machine Learning Classification

Multi-label Learning Trees, Embeddings, and much more! Purushottam Kar SIGML Department of CSE IIT Kanpur Special Interest Group in Machine Learning Classification Paradigms Pick one Pick one Pick all applicable Label 1 Label 1 Label 1

839 views • 32 slides
Lecture 18: Semantic Role Labeling &amp; Semantic Parsing Kai-Wei Chang CS @ University of

Lecture 18: Semantic Role Labeling &amp; Semantic Parsing Kai-Wei Chang CS @ University of

Lecture 18: Semantic Role Labeling &amp; Semantic Parsing Kai-Wei Chang CS @ University of Virginia kw@kwchang.net Couse webpage: http://kwchang.net/teaching/NLP16 CS6501-NLP 1 Computational Semantics v Many high-level applications v

389 views • 15 slides
Skolem labelled graphs, old and new results Nabil Shalaby Department of Mathematics and

Skolem labelled graphs, old and new results Nabil Shalaby Department of Mathematics and

Skolem labelled graphs, old and new results Nabil Shalaby Department of Mathematics and Statistics Memorial University of Newfoundland This is joint work with David Pike and Asiyeh Sanaei CanaDAM 2013 Nabil Shalaby (MUN) Skolem labelled

756 views • 41 slides

More recommend