bertscore evaluating text generation with bert
play

BERTScore: Evaluating Text Generation with BERT Varsha Kishore - PowerPoint PPT Presentation

Mar 26, 2023 •780 likes •1.02k views

BERTScore: Evaluating Text Generation with BERT Varsha Kishore Tianyi Zhang Felix Wu Kilian Q. Weinberger Yoav Artzi I am like I like translate ich liebe es I like it I love it I am loving it I am like Candidate Reference I like

  1. BERTScore: Evaluating Text Generation with BERT Varsha Kishore Tianyi Zhang Felix Wu Kilian Q. Weinberger Yoav Artzi

  2. I am like I like translate ich liebe es I like it I love it I am loving it

  3. I am like Candidate Reference I like translate ich liebe es I like it I love it I love it I am loving it 0.88/1.00 Metric

  4. Text Generation Evaluation Metrics N-gram matching Embedding-based approaches metrics BLEU (Papineni et al., 2002) Meant 2.0 (Lo, 2017) METEOR (Banerjee & Lavie, 2005) YiSi -1 (Lo et al., 2018) ROUGE (Lin, 2004) BERTScore chrF (Popovic, 2015)

  5. BLEU N-gram Matching Reference 
 The weather is cold today Candidate 1 
 Candidate 2 
 The weather is sunny today It is freezing today BLEU cannot identify synonyms BLEU gives higher score to candidate 1

  6. BERTScore: an evaluation metric that uses BERT embeddings

  7. BERT Transformer model pre-trained on 
 masked language modeling and next sentence prediction Generates word token embeddings that reflect their context

  8. BERTScore the weather is cold today Reference the weather is cold today Pairwise cosine similarity Candidate it is freezing today Contextual it is freezing today embedding

  9. Greedy Matching the weather Candidate is ≈ ç cold today is freezing today it ≈ ç Reference

  10. Greedy Matching Precision Recall Match words in candidate to reference Match words in reference to candidate

  11. Greedy Matching Precision Recall Match words in candidate to reference Match words in reference to candidate

  12. Greedy Matching Precision Recall Match words in candidate to reference Match words in reference to candidate

  13. Greedy Matching Precision Recall Match words in candidate to reference Match words in reference to candidate

  14. Greedy Matching Precision Recall 0.713 0.713 0.515 0.858 0.858 0.796 0.796 0.913 0.913 Match words in candidate to reference Match words in reference to candidate

  15. Greedy Matching - Aggregate Precision Recall 0.713 0.515 0.858 0.796 0.713 0.858 0.796 0.913 0.913

  16. Greedy Matching - Aggregate Precision Recall 0.713 0.515 0.858 0.796 0.713 0.858 0.796 0.913 0.913 0.759 0.820

  17. <latexit sha1_base64="ft0u0d5gLkLIktmgMFQEAFxGQ3o=">ACRnicbZDNaxNBGMbfTW0bo61rPXoZDIghN1SUJBCQCgeo5gPyC5hdvJuMmT2g5l3S8Oy4P/mpWdv/glePLSIV2ezOWjiCwMPz/O8zMwvypU05HnfndbBg8Oj4/bDzqPHJ6dP3KdnI5MVWuBQZCrTk4gbVDLFIUlSOMk18iRSOI5W7+t8fI3ayCz9TOscw4QvUhlLwclaMzcMCG+ovPIrdsnOWfAuiDUXZeMONApZr1YsEPOMWGN/QsGVqr91udxsztej1vM2xf+FvRhe0MZu63YJ6JIsGUhOLGTH0vp7DkmqRQWHWCwmDOxYovcGplyhM0YbnBULGX1pmzONP2pMQ27t8bJU+MWSeRbSaclmY3q83/ZdOC4rdhKdO8IExFc1FcKEYZq5myubQISK2t4EJL+1YmltxyJEu+YyH4u1/eF6Pznu/1/I8X3f7oS4OjDc/hBbwCH95AHz7AIYg4Cv8gDu4d26dn84v53dTbTlbhM/gn2nBHwO+s9M=</latexit> <latexit sha1_base64="ft0u0d5gLkLIktmgMFQEAFxGQ3o=">ACRnicbZDNaxNBGMbfTW0bo61rPXoZDIghN1SUJBCQCgeo5gPyC5hdvJuMmT2g5l3S8Oy4P/mpWdv/glePLSIV2ezOWjiCwMPz/O8zMwvypU05HnfndbBg8Oj4/bDzqPHJ6dP3KdnI5MVWuBQZCrTk4gbVDLFIUlSOMk18iRSOI5W7+t8fI3ayCz9TOscw4QvUhlLwclaMzcMCG+ovPIrdsnOWfAuiDUXZeMONApZr1YsEPOMWGN/QsGVqr91udxsztej1vM2xf+FvRhe0MZu63YJ6JIsGUhOLGTH0vp7DkmqRQWHWCwmDOxYovcGplyhM0YbnBULGX1pmzONP2pMQ27t8bJU+MWSeRbSaclmY3q83/ZdOC4rdhKdO8IExFc1FcKEYZq5myubQISK2t4EJL+1YmltxyJEu+YyH4u1/eF6Pznu/1/I8X3f7oS4OjDc/hBbwCH95AHz7AIYg4Cv8gDu4d26dn84v53dTbTlbhM/gn2nBHwO+s9M=</latexit> <latexit sha1_base64="ft0u0d5gLkLIktmgMFQEAFxGQ3o=">ACRnicbZDNaxNBGMbfTW0bo61rPXoZDIghN1SUJBCQCgeo5gPyC5hdvJuMmT2g5l3S8Oy4P/mpWdv/glePLSIV2ezOWjiCwMPz/O8zMwvypU05HnfndbBg8Oj4/bDzqPHJ6dP3KdnI5MVWuBQZCrTk4gbVDLFIUlSOMk18iRSOI5W7+t8fI3ayCz9TOscw4QvUhlLwclaMzcMCG+ovPIrdsnOWfAuiDUXZeMONApZr1YsEPOMWGN/QsGVqr91udxsztej1vM2xf+FvRhe0MZu63YJ6JIsGUhOLGTH0vp7DkmqRQWHWCwmDOxYovcGplyhM0YbnBULGX1pmzONP2pMQ27t8bJU+MWSeRbSaclmY3q83/ZdOC4rdhKdO8IExFc1FcKEYZq5myubQISK2t4EJL+1YmltxyJEu+YyH4u1/eF6Pznu/1/I8X3f7oS4OjDc/hBbwCH95AHz7AIYg4Cv8gDu4d26dn84v53dTbTlbhM/gn2nBHwO+s9M=</latexit> <latexit sha1_base64="ft0u0d5gLkLIktmgMFQEAFxGQ3o=">ACRnicbZDNaxNBGMbfTW0bo61rPXoZDIghN1SUJBCQCgeo5gPyC5hdvJuMmT2g5l3S8Oy4P/mpWdv/glePLSIV2ezOWjiCwMPz/O8zMwvypU05HnfndbBg8Oj4/bDzqPHJ6dP3KdnI5MVWuBQZCrTk4gbVDLFIUlSOMk18iRSOI5W7+t8fI3ayCz9TOscw4QvUhlLwclaMzcMCG+ovPIrdsnOWfAuiDUXZeMONApZr1YsEPOMWGN/QsGVqr91udxsztej1vM2xf+FvRhe0MZu63YJ6JIsGUhOLGTH0vp7DkmqRQWHWCwmDOxYovcGplyhM0YbnBULGX1pmzONP2pMQ27t8bJU+MWSeRbSaclmY3q83/ZdOC4rdhKdO8IExFc1FcKEYZq5myubQISK2t4EJL+1YmltxyJEu+YyH4u1/eF6Pznu/1/I8X3f7oS4OjDc/hBbwCH95AHz7AIYg4Cv8gDu4d26dn84v53dTbTlbhM/gn2nBHwO+s9M=</latexit> F1 = 2 Precision · Recall Precision + Recall

  18. Reference the weather is cold today F1 Score Candidate it is freezing today Pairwise Contextual cosine embedding similarity

  19. Evaluation: WMT Translation Benchmark Human Metric Reference: The weather is cold today. 0.85 0.77 Candidate: It is freezing today. compute correlation Reference: The garden is nice. 0.77 0.71 Candidate: The garden was pretty. Reference: I like apples very much. 0.80 0.79 Candidate: I love apples.

  20. Correlation Study 0.8 BLEU ITER YiSi-1 RUSE BertScore F1 0.6 Correlation 0.4 0.2 0 Czech-English German-English English-Czech English-German Language Pair

  21. 4 tasks 8 languages 363 systems

  22. Download here :https://pypi.org/project/bert-score/ Or Just: pip install bert_score Github

Download Presentation
Download Policy: The content available on the website is offered to you 'AS IS' for your personal information and use only. It cannot be commercialized, licensed, or distributed on other websites without prior consent from the author. To download a presentation, simply click this link. If you encounter any difficulties during the download process, it's possible that the publisher has removed the file from their server.

Recommend

What can Statistical Machine Translation teach Neural Text Generation about Optimization? Graham

What can Statistical Machine Translation teach Neural Text Generation about Optimization? Graham

What can Statistical Machine Translation teach Neural Text Generation about Optimization? Graham Neubig @ NAACL Workshop on Methods for Optimizing and Evaluating Neural Language Generation 6/6/2019 or How to Optimize your Neural Generation

638 views • 34 slides
GANocracy Outline Background: Text Generation Latent-Variable Generation Learning

GANocracy Outline Background: Text Generation Latent-Variable Generation Learning

Controlling Text Generation Alexander Rush (and Sam Wiseman) Harvard / Cornell Tech GANocracy Outline Background: Text Generation Latent-Variable Generation Learning Neural Templates Machine Learning for Text Generation y 1: T =

710 views • 55 slides
Text-to-Text Generation Katja Filippova katjaf@google.com Friday, August 19, 2011 1 This

Text-to-Text Generation Katja Filippova katjaf@google.com Friday, August 19, 2011 1 This

Text-to-Text Generation Katja Filippova katjaf@google.com Friday, August 19, 2011 1 This course A quick overview of a number of topics under the umbrella term text-to-text generation. Research problems - what is being done and

1.95k views • 119 slides
BERT 3.0 The New BERT Wheres Ernie????? Logging into Bert BERT now uses the same style logon as

BERT 3.0 The New BERT Wheres Ernie????? Logging into Bert BERT now uses the same style logon as

BERT 3.0 The New BERT Wheres Ernie????? Logging into Bert BERT now uses the same style logon as the other SEIMS+ applications. BERT Main Screen Icons on Main Screen of BERT There are several icons on the main screen for BERT. These icons are

590 views • 16 slides
Unsupervised Machine Translation Sachin Kumar Conditional Text Generation Generate text

Unsupervised Machine Translation Sachin Kumar Conditional Text Generation Generate text

CMU CS11-737: Multilingual NLP (Fall 2020) Unsupervised Machine Translation Sachin Kumar Conditional Text Generation Generate text according to a specification: P(Y|X) Input X Output Y (Text) Task English Hindi Machine Translation

743 views • 35 slides
11-830 Computational Ethics for NLP Ethical Concerns on OpenAI Text Generation System Discussion

11-830 Computational Ethics for NLP Ethical Concerns on OpenAI Text Generation System Discussion

11-830 Computational Ethics for NLP Ethical Concerns on OpenAI Text Generation System Discussion OpenAI Text Generation System CNN: This AI is so good at writing that its creators wont let you use it Takes in a text prompt and

336 views • 8 slides
BERT Bidirectional Encoder Representations from Transformers Introduction What is BERT?

BERT Bidirectional Encoder Representations from Transformers Introduction What is BERT?

BERT Bidirectional Encoder Representations from Transformers Introduction What is BERT? Latest language representational model BERT is conceptually simple and empirically powerful. One of the biggest challenges in natural language

839 views • 20 slides
Text-to-Image Generation Yu Cheng Text-to-Image Synthesis Text-to-Image Synthesis

Text-to-Image Generation Yu Cheng Text-to-Image Synthesis Text-to-Image Synthesis

Text-to-Image Generation Yu Cheng Text-to-Image Synthesis Text-to-Image Synthesis StackGAN, AttnGAN, TAGAN, ObjGAN Text-to-Video Synthesis GAN-based methods, VAE-based methods, StoryGAN Dialogue-based Image Synthesis

827 views • 45 slides
From Text Message to Research Paper: Teaching the Text Message Generation Heidi Wright Ohio

From Text Message to Research Paper: Teaching the Text Message Generation Heidi Wright Ohio

From Text Message to Research Paper: Teaching the Text Message Generation Heidi Wright Ohio State University Introduction and Organization Origins- SIUC 2006 Increased number of tasks and shorter time frame Insufficient background

241 views • 22 slides
Parsing Eric McCreath Overview In this lecture we will look at: structured text, generation,

Parsing Eric McCreath Overview In this lecture we will look at: structured text, generation,

Parsing Eric McCreath Overview In this lecture we will look at: structured text, generation, parsing, tokens, grammars, and writing a simple parser. 2 Structured Text Information is often stored in text files. Also we often provide

522 views • 13 slides
Non-Monotonic Sequential Text Generation Sean Welleck, Kiant Brantley, Hal Daum III,

Non-Monotonic Sequential Text Generation Sean Welleck, Kiant Brantley, Hal Daum III,

Non-Monotonic Sequential Text Generation Sean Welleck, Kiant Brantley, Hal Daum III, Kyunghyun Cho Sequential Text Generation Y = ( y 1 , y 2 , , y N ) ( hi , how , are , you , ? ) Sequential Text Generation Unconditional Y ( hi ,

459 views • 24 slides
Statistical Perspectives on Text-to-Text Generation Noah Smith Language Technologies Institute

Statistical Perspectives on Text-to-Text Generation Noah Smith Language Technologies Institute

Statistical Perspectives on Text-to-Text Generation Noah Smith Language Technologies Institute Machine Learning Department School of Computer Science Carnegie Mellon University nasmith@cs.cmu.edu Im A Learning Guy I use statistics for

1.2k views • 82 slides
Text Generation with Exemplar-based Adaptive Decoding Hao Peng, Ankur Parikh, Manaal Faruqui,

Text Generation with Exemplar-based Adaptive Decoding Hao Peng, Ankur Parikh, Manaal Faruqui,

Text Generation with Exemplar-based Adaptive Decoding Hao Peng, Ankur Parikh, Manaal Faruqui, Bhuwan Dhingra, Dipanjan Das @NAACL June 4, 2019 Outline Background and Overview Adaptive Decoding Experiments Conditioned Text Generation

652 views • 49 slides
Speech Generation From Concept and from Text Martin Jansche CS 6998 2004-02-11 Components of

Speech Generation From Concept and from Text Martin Jansche CS 6998 2004-02-11 Components of

Speech Generation From Concept and from Text Martin Jansche CS 6998 2004-02-11 Components of spoken output systems Front end: From input to control parameters. From naturally occurring text; or From constrained mark-up language; or

420 views • 27 slides
Intractable Peacebuilding: Evaluating a Generation of Work Across the Israeli-Palestinian Divide

Intractable Peacebuilding: Evaluating a Generation of Work Across the Israeli-Palestinian Divide

Intractable Peacebuilding: Evaluating a Generation of Work Across the Israeli-Palestinian Divide Ned Lazarus Network for Peacebuilding Evaluation Thursday Talk May 21, 2015 Presentation Research Overview: Four Evaluative Studies

471 views • 17 slides
Automatically Evaluating Text Coherence Using Discourse Relations Ziheng Lin , Hwee Tou Ng and

Automatically Evaluating Text Coherence Using Discourse Relations Ziheng Lin , Hwee Tou Ng and

Automatically Evaluating Text Coherence Using Discourse Relations Ziheng Lin , Hwee Tou Ng and Min-Yen Kan Department of Computer Science National University of Singapore Introduction Textual coherence discourse structure Conditional

702 views • 36 slides
Ruiqi Guo, Philip Sun, Erik Lindgren, Quan Geng, David Simcha, Felix Chern, Sanjiv Kumar Overview

Ruiqi Guo, Philip Sun, Erik Lindgren, Quan Geng, David Simcha, Felix Chern, Sanjiv Kumar Overview

Ruiqi Guo, Philip Sun, Erik Lindgren, Quan Geng, David Simcha, Felix Chern, Sanjiv Kumar Overview Application: recommender systems Application: recommender systems Application: recommender systems Application: recommender systems Application:

761 views • 38 slides
Binomial Representation Theorem Recall the discrete stochastic integral : if { X n } n 0 is

Binomial Representation Theorem Recall the discrete stochastic integral : if { X n } n 0 is

Binomial Representation Theorem Recall the discrete stochastic integral : if { X n } n 0 is a ( P , {F n } n 0 )- martingale and { n } n 1 is {F n } n 0 -previsible, then n 1 Z n = Z 0 + j +1 X j +1 X

790 views • 12 slides
Double integrals First, let's recall Riemann sums for integrals with a single variable... 1 2

Double integrals First, let's recall Riemann sums for integrals with a single variable... 1 2

Double integrals First, let's recall Riemann sums for integrals with a single variable... 1 2 3 4 5 Some comments: (1) Note that we can estimate double integrals using double Riemann sums. While this is tedious by hand, computers do this

208 views • 19 slides
Refresh Your Knowledge 7 Select all that are true about policy gradients: V ( ) = E

Refresh Your Knowledge 7 Select all that are true about policy gradients: V ( ) = E

Lecture 9: Policy Gradient II 1 Emma Brunskill CS234 Reinforcement Learning. Winter 2020 Additional reading: Sutton and Barto 2018 Chp. 13 1 With many slides from or derived from David Silver and John Schulman and Pieter Abbeel Lecture 9: Policy

870 views • 59 slides
Computational Model of False Recall Rishabh Nigam 10598 Cognitive Sciences Introduction

Computational Model of False Recall Rishabh Nigam 10598 Cognitive Sciences Introduction

Computational Model of False Recall Rishabh Nigam 10598 Cognitive Sciences Introduction False Recall - &gt; remembering events that didnt happened. Deese demonstrated this using a word list in 1959. Experiment: The participants

475 views • 12 slides
Recall: Liang Barsky 3D Clipping Goal: Clip object edge-by-edge against Canonical View

Recall: Liang Barsky 3D Clipping Goal: Clip object edge-by-edge against Canonical View

Recall: Liang Barsky 3D Clipping Goal: Clip object edge-by-edge against Canonical View volume (CVV) Problem: 2 end-points of edge: A = (Ax, Ay, Az, Aw) and C = (Cx, Cy, Cz, Cw) If edge intersects with CVV, compute intersection

904 views • 44 slides
CS70: Jean Walrand: Lecture 24. Recall definition: Conditional Probability. More fun with

CS70: Jean Walrand: Lecture 24. Recall definition: Conditional Probability. More fun with

CS70: Jean Walrand: Lecture 24. Recall definition: Conditional Probability. More fun with conditional probability. Toss a red and a blue die, sum is 4, Consider = { 1 , 2 ,..., N } with Pr [ n ] = p n . What is probability that red is 1?

296 views • 5 slides
Backward Induction in Games without Perfect Recall John Hillas and Dmitriy Kvasov CMSS Workshop,

Backward Induction in Games without Perfect Recall John Hillas and Dmitriy Kvasov CMSS Workshop,

Backward Induction in Games without Perfect Recall John Hillas and Dmitriy Kvasov CMSS Workshop, February 2012 Backward Induction The game theorists who defined the equilibrium concepts that we now think of as various forms of backwards

579 views • 24 slides

More recommend