SLIDE 1
Understanding Task Design Trade-offs in Crowdsourced Paraphrase Collection
Youxuan Lucy Jiang, Jonathan Kummerfeld and Walter Lasecki
Paraphrases are useful across a wide range of NLP tasks
Motivation Experiment Results
Understanding Task Design Trade-offs in Crowdsourced Paraphrase - - PowerPoint PPT Presentation
Understanding Task Design Trade-offs in Crowdsourced Paraphrase - - PowerPoint PPT Presentation
Understanding Task Design Trade-offs in Crowdsourced Paraphrase Collection Youxuan Lucy Jiang, Jonathan Kummerfeld and Walter Lasecki Paraphrases are useful across Motivation Experiment a wide range of NLP tasks Results
SLIDE 2
SLIDE 3
Paraphrases are useful across a wide range of NLP tasks
Motivation Experiment Results
Sentence Paraphrase 1 Paraphrase 2 …
SLIDE 4
Paraphrases are useful across a wide range of NLP tasks
Motivation Experiment Results
Sentence Paraphrase 1 Paraphrase 2 …
SLIDE 5
Paraphrases are useful across a wide range of NLP tasks
Motivation Experiment Results
Sentence Paraphrase 1 Paraphrase 2 … Structured Output
SLIDE 6
Paraphrases are useful across a wide range of NLP tasks
Motivation Experiment Results
The boy desires the girl to believe him. The boy desires to be believed by the girl. The boy has a desire to be believed by the girl. The boy’s desire is for the girl to believe him. …
SLIDE 7
Paraphrases are useful across a wide range of NLP tasks
Motivation Experiment Results
What is the smallest state by area? Which state has the least area? Name the state with the smallest area … λx1.smallest(x2, λx1.state(x1), λx1.λx2.area(x1,x2))
SLIDE 8
Paraphrases are useful across a wide range of NLP tasks
Motivation Experiment Results
What is the smallest state by area? Which state has the least area? Name the state with the smallest area … SELECT state.state_name FROM state WHERE state.area = ( SELECT MIN (state.area) FROM state ) ;
SLIDE 9
Paraphrases are useful across a wide range of NLP tasks
Motivation Experiment Results
What is the smallest state by area? Which state has the least area? Name the state with the smallest area … SELECT state.state_name FROM state WHERE state.area = ( SELECT MIN (state.area) FROM state ) ;
SLIDE 10
Many factors affect the quality and diversity of crowdsourced paraphrases
Motivation Experiment Results
SLIDE 11
Many factors affect the quality and diversity of crowdsourced paraphrases
Motivation Experiment Results
SLIDE 12
Many factors affect the quality and diversity of crowdsourced paraphrases
Motivation Experiment Results
Title
SLIDE 13
Many factors affect the quality and diversity of crowdsourced paraphrases
Motivation Experiment Results
Title Instructions …
SLIDE 14
Many factors affect the quality and diversity of crowdsourced paraphrases
Motivation Experiment Results
Title Instructions … Examples
SLIDE 15
Many factors affect the quality and diversity of crowdsourced paraphrases
Motivation Experiment Results
Title Instructions … Examples Prompt Sentence
SLIDE 16
Many factors affect the quality and diversity of crowdsourced paraphrases
Motivation Experiment Results
Title Instructions … Examples Prompt Sentence
SLIDE 17
Many factors affect the quality and diversity of crowdsourced paraphrases
Motivation Experiment Results
Title Instructions … Examples Prompt Sentence
SLIDE 18
Instructions … Title Prompt Sentence
Many factors affect the quality and diversity of crowdsourced paraphrases
Motivation Experiment Results
Examples
SLIDE 19
Examples Instructions … Title Prompt Sentence
Many factors affect the quality and diversity of crowdsourced paraphrases
Motivation Experiment Results
SLIDE 20
Title Examples Prompt Sentence
Many factors affect the quality and diversity of crowdsourced paraphrases
Motivation Experiment Results
Instructions …
SLIDE 21
Title Examples Prompt Sentence
Many factors affect the quality and diversity of crowdsourced paraphrases
Motivation Experiment Results
Instructions … $0.05 - participation
SLIDE 22
Title Examples Prompt Sentence
Many factors affect the quality and diversity of crowdsourced paraphrases
Motivation Experiment Results
Instructions … $0.05 - participation $0.05 - bonus
SLIDE 23
Examples Instructions … Title
Many factors affect the quality and diversity of crowdsourced paraphrases
Motivation Experiment Results
Prompt Sentence
SLIDE 24
Many factors affect the quality and diversity of crowdsourced paraphrases
Motivation Experiment Results
SLIDE 25
Many factors affect the quality and diversity of crowdsourced paraphrases
Motivation Experiment Results
SLIDE 26
Many factors affect the quality and diversity of crowdsourced paraphrases
Motivation Experiment Results
SLIDE 27
Many factors affect the quality and diversity of crowdsourced paraphrases
Motivation Experiment Results
SLIDE 28
Many factors affect the quality and diversity of crowdsourced paraphrases
Motivation Experiment Results
SLIDE 29
Many factors affect the quality and diversity of crowdsourced paraphrases
Motivation Experiment Results
SLIDE 30
Many factors affect the quality and diversity of crowdsourced paraphrases
Motivation Experiment Results
SLIDE 31
Many factors affect the quality and diversity of crowdsourced paraphrases
Motivation Experiment Results
SLIDE 32
Many factors affect the quality and diversity of crowdsourced paraphrases
Motivation Experiment Results
SLIDE 33
Many factors affect the quality and diversity of crowdsourced paraphrases
Motivation Experiment Results
SLIDE 34
Many factors affect the quality and diversity of crowdsourced paraphrases
Motivation Experiment Results
SLIDE 35
Motivation Experiment Results
SLIDE 36
Motivation Experiment Results
50 60 70 80 90 100 Syntactic Lexical Variation Lexical & Syntactic No Examples
Correctness Diversity
SLIDE 37
Motivation Experiment Results
50 60 70 80 90 100 Syntactic Lexical Variation Lexical & Syntactic No Examples Correctness Diversity
SLIDE 38
Motivation Experiment Results
50 60 70 80 90 100 Syntactic Lexical Variation Lexical & Syntactic No Examples Correctness Diversity
SLIDE 39
Priming has a significant impact
Motivation Experiment Results
50 60 70 80 90 100 Syntactic Lexical Variation Lexical & Syntactic No Examples Correctness Diversity
SLIDE 40
Priming has a significant impact
Motivation Experiment Results
50 60 70 80 90 100 Syntactic Lexical Variation Lexical & Syntactic No Examples Correctness Diversity
SLIDE 41
Priming has a significant impact
Motivation Experiment Results
50 60 70 80 90 100 Single Source Chain Answers Correctness Diversity
SLIDE 42
Priming has a significant impact
Motivation Experiment Results
50 60 70 80 90 100 Single Source Chain Answers Correctness Diversity
SLIDE 43
Priming has a significant impact
Motivation Experiment Results
50 60 70 80 90 100 Single Source Chain Answers Correctness Diversity
SLIDE 44
Jargon is a challenge
Motivation Experiment Results
30 40 50 60 70 80 90 100 Course Questions Advising Dialogue GeoQuery WSJ Ubuntu IRC Correctness Diversity 30 100
SLIDE 45
Jargon is a challenge
Motivation Experiment Results
30 40 50 60 70 80 90 100 Course Questions Advising Dialogue GeoQuery WSJ Ubuntu IRC Correctness Diversity 30 100
SLIDE 46
Jargon is a challenge
Motivation Experiment Results
30 40 50 60 70 80 90 100 Course Questions Advising Dialogue GeoQuery WSJ Ubuntu IRC Correctness Diversity 30 100
SLIDE 47
The first systematic study of trade-offs when crowdsourcing paraphrases
Motivation Experiment Results Youxuan Lucy Jiang, Jonathan Kummerfeld and Walter Lasecki Supported by IBM