understanding task design trade offs in crowdsourced
play

Understanding Task Design Trade-offs in Crowdsourced Paraphrase - PowerPoint PPT Presentation

Feb 10, 2023 •149 likes •624 views

Understanding Task Design Trade-offs in Crowdsourced Paraphrase Collection Youxuan Lucy Jiang, Jonathan Kummerfeld and Walter Lasecki Paraphrases are useful across Motivation Experiment a wide range of NLP tasks Results

  1. Understanding Task Design Trade-offs in Crowdsourced Paraphrase Collection Youxuan Lucy Jiang, Jonathan Kummerfeld and 
 Walter Lasecki

  2. Paraphrases are useful across 
 Motivation 
 Experiment 
 a wide range of NLP tasks Results

  3. 
 
 
 Paraphrases are useful across 
 Motivation 
 Experiment 
 a wide range of NLP tasks Results Sentence 
 Paraphrase 1 
 Paraphrase 2 
 …

  4. 
 
 
 Paraphrases are useful across 
 Motivation 
 Experiment 
 a wide range of NLP tasks Results Sentence 
 Paraphrase 1 
 Paraphrase 2 
 …

  5. 
 
 
 Paraphrases are useful across 
 Motivation 
 Experiment 
 a wide range of NLP tasks Results Sentence 
 Paraphrase 1 
 Structured Output Paraphrase 2 
 …

  6. Paraphrases are useful across 
 Motivation 
 Experiment 
 a wide range of NLP tasks Results The boy desires the girl 
 to believe him. The boy desires to be 
 believed by the girl. The boy has a desire to 
 be believed by the girl. The boy’s desire is for 
 the girl to believe him. …

  7. 
 
 
 Paraphrases are useful across 
 Motivation 
 Experiment 
 a wide range of NLP tasks Results What is the smallest 
 state by area? 
 λ x1.smallest(x2, 
 Which state has 
 λ x1.state(x1), 
 the least area? 
 λ x1. λ x2.area(x1,x2)) Name the state with 
 the smallest area 
 …

  8. 
 
 
 Paraphrases are useful across 
 Motivation 
 Experiment 
 a wide range of NLP tasks Results What is the smallest 
 state by area? 
 SELECT state.state_name 
 FROM state 
 Which state has 
 WHERE state.area = ( the least area? 
 SELECT MIN (state.area) FROM state Name the state with 
 ) ; the smallest area 
 …

  9. 
 
 
 Paraphrases are useful across 
 Motivation 
 Experiment 
 a wide range of NLP tasks Results What is the smallest 
 state by area? 
 SELECT state.state_name 
 FROM state 
 Which state has 
 WHERE state.area = ( the least area? 
 SELECT MIN (state.area) FROM state Name the state with 
 ) ; the smallest area 
 …

  10. Many factors affect the quality and Motivation 
 Experiment 
 diversity of crowdsourced paraphrases Results

  11. Many factors affect the quality and Motivation 
 Experiment 
 diversity of crowdsourced paraphrases Results

  12. Many factors affect the quality and Motivation 
 Experiment 
 diversity of crowdsourced paraphrases Results Title

  13. Many factors affect the quality and Motivation 
 Experiment 
 diversity of crowdsourced paraphrases Results Title Instructions …

  14. Many factors affect the quality and Motivation 
 Experiment 
 diversity of crowdsourced paraphrases Results Title Instructions … Examples

  15. Many factors affect the quality and Motivation 
 Experiment 
 diversity of crowdsourced paraphrases Results Title Instructions … Examples Prompt Sentence

  16. Many factors affect the quality and Motivation 
 Experiment 
 diversity of crowdsourced paraphrases Results Title Instructions … Examples Prompt Sentence

  17. Many factors affect the quality and Motivation 
 Experiment 
 diversity of crowdsourced paraphrases Results Title Instructions … Examples Prompt Sentence

  18. Many factors affect the quality and Motivation 
 Experiment 
 diversity of crowdsourced paraphrases Results Title Instructions … Examples Prompt Sentence

  19. Many factors affect the quality and Motivation 
 Experiment 
 diversity of crowdsourced paraphrases Results Title Instructions … Examples Prompt Sentence

  20. Many factors affect the quality and Motivation 
 Experiment 
 diversity of crowdsourced paraphrases Results Title Instructions … Examples Prompt Sentence

  21. Many factors affect the quality and Motivation 
 Experiment 
 diversity of crowdsourced paraphrases Results Title Instructions … $0.05 - participation Examples Prompt Sentence

  22. Many factors affect the quality and Motivation 
 Experiment 
 diversity of crowdsourced paraphrases Results Title Instructions … $0.05 - participation Examples $0.05 - bonus Prompt Sentence

  23. Many factors affect the quality and Motivation 
 Experiment 
 diversity of crowdsourced paraphrases Results Title Instructions … Examples Prompt Sentence

  24. Many factors affect the quality and Motivation 
 Experiment 
 diversity of crowdsourced paraphrases Results

  25. Many factors affect the quality and Motivation 
 Experiment 
 diversity of crowdsourced paraphrases Results

  26. Many factors affect the quality and Motivation 
 Experiment 
 diversity of crowdsourced paraphrases Results

  27. Many factors affect the quality and Motivation 
 Experiment 
 diversity of crowdsourced paraphrases Results

  28. Many factors affect the quality and Motivation 
 Experiment 
 diversity of crowdsourced paraphrases Results

  29. Many factors affect the quality and Motivation 
 Experiment 
 diversity of crowdsourced paraphrases Results

  30. Many factors affect the quality and Motivation 
 Experiment 
 diversity of crowdsourced paraphrases Results

  31. Many factors affect the quality and Motivation 
 Experiment 
 diversity of crowdsourced paraphrases Results

  32. Many factors affect the quality and Motivation 
 Experiment 
 diversity of crowdsourced paraphrases Results

  33. Many factors affect the quality and Motivation 
 Experiment 
 diversity of crowdsourced paraphrases Results

  34. Many factors affect the quality and Motivation 
 Experiment 
 diversity of crowdsourced paraphrases Results

  35. Motivation 
 Experiment 
 Results

  36. Motivation 
 Experiment 
 Results 100 90 80 Correctness 70 Diversity 60 50 Syntactic Lexical 
 Lexical & 
 No Examples Variation Syntactic

  37. Motivation 
 Experiment 
 Results 100 90 80 Correctness 70 Diversity 60 50 Syntactic Lexical 
 Lexical & 
 No Examples Variation Syntactic

  38. Motivation 
 Experiment 
 Results 100 90 80 Correctness 70 Diversity 60 50 Syntactic Lexical 
 Lexical & 
 No Examples Variation Syntactic

  39. Motivation 
 Priming has a significant impact Experiment 
 Results 100 90 80 Correctness 70 Diversity 60 50 Syntactic Lexical 
 Lexical & 
 No Examples Variation Syntactic

  40. Motivation 
 Priming has a significant impact Experiment 
 Results 100 90 80 Correctness 70 Diversity 60 50 Syntactic Lexical 
 Lexical & 
 No Examples Variation Syntactic

  41. Motivation 
 Priming has a significant impact Experiment 
 Results 100 90 80 Correctness 70 Diversity 60 50 Single 
 Chain Answers Source

  42. Motivation 
 Priming has a significant impact Experiment 
 Results 100 90 80 Correctness 70 Diversity 60 50 Single 
 Chain Answers Source

  43. Motivation 
 Priming has a significant impact Experiment 
 Results 100 90 80 Correctness 70 Diversity 60 50 Single 
 Chain Answers Source

  44. Motivation 
 Jargon is a challenge Experiment 
 Results 100 100 90 80 70 Correctness 60 Diversity 50 40 30 30 Course 
 Advising 
 GeoQuery WSJ Ubuntu 
 Questions Dialogue IRC

  45. Motivation 
 Jargon is a challenge Experiment 
 Results 100 100 90 80 70 Correctness 60 Diversity 50 40 30 30 Course 
 Advising 
 GeoQuery WSJ Ubuntu 
 Questions Dialogue IRC

  46. Motivation 
 Jargon is a challenge Experiment 
 Results 100 100 90 80 70 Correctness 60 Diversity 50 40 30 30 Course 
 Advising 
 GeoQuery WSJ Ubuntu 
 Questions Dialogue IRC

  47. The first systematic study of trade-offs Motivation 
 Experiment 
 when crowdsourcing paraphrases Results Thank you! Youxuan Lucy Jiang, Jonathan Kummerfeld and 
 Walter Lasecki 
 Supported by IBM

Download Presentation
Download Policy: The content available on the website is offered to you 'AS IS' for your personal information and use only. It cannot be commercialized, licensed, or distributed on other websites without prior consent from the author. To download a presentation, simply click this link. If you encounter any difficulties during the download process, it's possible that the publisher has removed the file from their server.

Recommend

TRADE-OFFS AMONG AI TRADE-OFFS AMONG AI TECHNIQUES TECHNIQUES Christian Kaestner With slides

TRADE-OFFS AMONG AI TRADE-OFFS AMONG AI TECHNIQUES TECHNIQUES Christian Kaestner With slides

TRADE-OFFS AMONG AI TRADE-OFFS AMONG AI TECHNIQUES TECHNIQUES Christian Kaestner With slides adopted from Eunsuk Kang Required reading: Vogelsang, Andreas, and Markus Borg. " Requirements Engineering for Machine Learning:

760 views • 54 slides
Chapter 2 Trade-offs, Comparative Advantage, and the Market System Modeling Trade-offs:

Chapter 2 Trade-offs, Comparative Advantage, and the Market System Modeling Trade-offs:

Chapter 2 Trade-offs, Comparative Advantage, and the Market System Modeling Trade-offs: Production Possibilities Frontier (PPF) The PPF: Demonstrates the maximum attainable combination of two products with a given technology and a given

592 views • 13 slides
Markov chaining Checkout DataStructures project from SVN Understanding the engineering trade-offs

Markov chaining Checkout DataStructures project from SVN Understanding the engineering trade-offs

Data-structure-palooza Fixed-length queues Markov chaining Checkout DataStructures project from SVN Understanding the engineering trade-offs when storing data Boil down data types (e.g., lists) to their essential operations Choosing a

602 views • 20 slides
Checkout DataStructures from SVN Understanding the engineering trade-offs when storing data

Checkout DataStructures from SVN Understanding the engineering trade-offs when storing data

Data-structure-palooza Checkout DataStructures from SVN Understanding the engineering trade-offs when storing data Boil down data types (e.g., lists) to their essential operations Choosing a data structure for a project then becomes:

664 views • 14 slides
Data Structures + BIG-O Notation Understanding the engineering trade-offs when storing data

Data Structures + BIG-O Notation Understanding the engineering trade-offs when storing data

CSSE 220 Data Structures + BIG-O Notation Understanding the engineering trade-offs when storing data Checkout LinkedListSimple project from public git Checkout SinglyLinkedList homework from git Data Structures Efficient ways to store data

239 views • 9 slides
Energy-efficient Computing for Wildlife Tracking Design Trade-offs and Early Experiences with

Energy-efficient Computing for Wildlife Tracking Design Trade-offs and Early Experiences with

Energy-efficient Computing for Wildlife Tracking Design Trade-offs and Early Experiences with ZebraNet Philo J uang, Hidekazu Oki Yong Wang, Margaret Martonosi Li-Shiuan Peh, Dan Rubenstein Princeton University CS Z25/4C38: Mobile and

503 views • 37 slides
Trade Offs in the Design of a Router with Both Guaranteed and Best-Effort Services for Networks

Trade Offs in the Design of a Router with Both Guaranteed and Best-Effort Services for Networks

Trade Offs in the Design of a Router with Both Guaranteed and Best-Effort Services for Networks on Chip E. Rijpkema, K. Goossens, A. R dulescu, J. Dielissen, J. van Meerbergen, P. Wielage, and E. Waterlander why Networks-on-Chip problems

523 views • 23 slides
Design and Performance Trade-offs in Parallelized RF SDR Architecture Ville Saari 1 arssinen 2

Design and Performance Trade-offs in Parallelized RF SDR Architecture Ville Saari 1 arssinen 2

Design and Performance Trade-offs in Parallelized RF SDR Architecture Ville Saari 1 arssinen 2 Sami Kiminki Aarno P Antti Immonen 2 Vesa Hirvisalo Jussi Ryyn anen Tommi Zetterman Aalto University Aalto University Nokia Research Center

600 views • 17 slides
History of Operating Systems What drives these trade-offs? Hardware User Applications

History of Operating Systems What drives these trade-offs? Hardware User Applications

10/1/16 COMP 530: Operating Systems COMP 530: Operating Systems Natural Selection Almost all OS design is about trade-offs History of Operating Systems What drives these trade-offs? Hardware User Applications Observation:

381 views • 6 slides
Time-memory Trade-offs for Near-collisions Conclusion Combining trunc & codes Time-memory

Time-memory Trade-offs for Near-collisions Conclusion Combining trunc & codes Time-memory

Introduction 1/24 Gatan Leurent Time-memory Trade-offs for Near-collisions Conclusion Combining trunc & codes Time-memory trade-offs Memoryless FSE 2013 UCL Crypto Group FSE 2013 Time-memory Trade-offs for Near-collisions G.

430 views • 31 slides
CrowdQ: Crowdsourced Query Understanding Gianluca Demar8ni, Beth

CrowdQ: Crowdsourced Query Understanding Gianluca Demar8ni, Beth

CrowdQ: Crowdsourced Query Understanding Gianluca Demar8ni, Beth Trushkowsky, Tim Kraska, Michael J. Franklin Scenario Find the birthdate of the mayor of

316 views • 15 slides
Extreme-scale Data Resilience Trade-offs at Experimental Facilities Sadaf Alam Chief

Extreme-scale Data Resilience Trade-offs at Experimental Facilities Sadaf Alam Chief

Extreme-scale Data Resilience Trade-offs at Experimental Facilities Sadaf Alam Chief Technology O ffi cer Swiss National Supercomputing Centre MSST (May 22, 2019) Outline Background Users, customers and services Co-design,

441 views • 25 slides
Enemy targeting, trade - offs, and the evolutionary Assembly of

Enemy targeting, trade - offs, and the evolutionary Assembly of

Enemy targeting, trade - offs, and the evolutionary Assembly of a tortoise beetle defense arsenal Fredric V. Vencl Ecology and Evolution, Stony Brook

262 views • 9 slides
Filling in the Blanks in Understanding Discourse Adverbials: Consistency, Conflict,

Filling in the Blanks in Understanding Discourse Adverbials: Consistency, Conflict,

Filling in the Blanks in Understanding Discourse Adverbials: Consistency, Conflict, and Context-Dependence in a Crowdsourced Elicitation Task Hannah Rohde, Anna Dickinson, Nathan Schneider, Christopher N. L. Clark, Annie Louis, &

818 views • 23 slides
Understand the trade-offs using compilers for Java applications (From AOT to JIT and Beyond!)

Understand the trade-offs using compilers for Java applications (From AOT to JIT and Beyond!)

Understand the trade-offs using compilers for Java applications (From AOT to JIT and Beyond!) Mark Stoodley Eclipse OpenJ9 & OMR project co-lead Senior Software Developer @ IBM Canada mstoodle@ca.ibm.com @mstoodle Java ecosystem has a

827 views • 56 slides
Object/Class Design Slides adapted from Craig Zilles 1 Design is hard Design is an art, not a

Object/Class Design Slides adapted from Craig Zilles 1 Design is hard Design is an art, not a

Object/Class Design Slides adapted from Craig Zilles 1 Design is hard Design is an art, not a science Large/infinite design space, not enumerable Requirements, constraints, trade-offs, and priorities You get better with practice /

417 views • 15 slides
What d do Employe yees W Want A Anywa yway? y? Presented By: Shameria Davis, Wellness

What d do Employe yees W Want A Anywa yway? y? Presented By: Shameria Davis, Wellness

What d do Employe yees W Want A Anywa yway? y? Presented By: Shameria Davis, Wellness Consultant I. Employee Motivation: Are your OBJECTIVES employees motivated or not? II. Employee Motivation: What To be inspired is helps when they're

282 views • 27 slides
The Supervisor as Motivator Presented by Human Resource Services rev. March 2011 Objectives

The Supervisor as Motivator Presented by Human Resource Services rev. March 2011 Objectives

The Supervisor as Motivator Presented by Human Resource Services rev. March 2011 Objectives 1) Define motivation. 2) Explain the supervisors role in motivating others. 3) Identify the three determinants of behavior and your role in

397 views • 10 slides
What is the purpose of the Barker Institute? Dr Brad Merrick - Director of 10 top tips the

What is the purpose of the Barker Institute? Dr Brad Merrick - Director of 10 top tips the

What is the purpose of the Barker Institute? Dr Brad Merrick - Director of 10 top tips the Barker Institute @bradmerrick for motivation and @barkerinstitute success! www.barkerinstitute.com.au brad@barkerinstitute.com.au SELF REFLECTION IS

485 views • 34 slides
Improving Student Engagement and Motivation at the High School and Middle School Level 1 Rob

Improving Student Engagement and Motivation at the High School and Middle School Level 1 Rob

MA MASA Oc Oct. 7, 7, 2019 2019 Improving Student Engagement and Motivation at the High School and Middle School Level 1 Rob Metz, Deputy Director, BARR Center Your MN Grown Presenters Bob Laney, BARR Trainer 2 Improving Student

198 views • 17 slides
SPECIAL OLYMPICS Volunteer Management Series VOLUNTEER MANAGEMENT SERIES F-1 Int roduc t ory

SPECIAL OLYMPICS Volunteer Management Series VOLUNTEER MANAGEMENT SERIES F-1 Int roduc t ory

SPECIAL OLYMPICS Volunteer Management Series VOLUNTEER MANAGEMENT SERIES F-1 Int roduc t ory Ex erc ise Volunteer Management Series Recall your motivations for volunteering with Special Olympics. Did the Program recognize and appreciate

351 views • 18 slides
Improving Student Engagement and Motivation through Practices of Assessment Wisconsin Mathematics

Improving Student Engagement and Motivation through Practices of Assessment Wisconsin Mathematics

MILWAUKEE MASTER TEACHER PARTNERSHIP Improving Student Engagement and Motivation through Practices of Assessment Wisconsin Mathematics Council May 4, 2018 358 Kern Boehr 11:30 am-12:30 pm Joan Masek Mary Zietlow Erin McReynolds Alexander

615 views • 32 slides
TA: Sara Guarino (e-mail: s.guarino@tcu.edu) TR, 2:00-3:20 pm, WIN 228 Objectives This is an

TA: Sara Guarino (e-mail: s.guarino@tcu.edu) TR, 2:00-3:20 pm, WIN 228 Objectives This is an

30433-EXPERIMENTAL PSYCHOLOGY: MOTIVATION Spring, 2020 Dr. Mauricio R. Papini (SWR 336, 817-257-6084, m.papini@tcu.edu) TA: Sara Guarino (e-mail: s.guarino@tcu.edu) TR, 2:00-3:20 pm, WIN 228 Objectives This is an introduction to the

411 views • 15 slides
Date 15.9.2015 O Date 15.9.2015 O INTERNATIONAL ELECTROTECHNICAL TRACKING SYSTEM

Date 15.9.2015 O Date 15.9.2015 O INTERNATIONAL ELECTROTECHNICAL TRACKING SYSTEM

Date 15.9.2015 O Date 15.9.2015 O INTERNATIONAL ELECTROTECHNICAL TRACKING SYSTEM http://www.iec.ch Date 15.9.2015 O WHO- REPORTING ON ROAD SAFETY: A QUIDE FOR JOURNALISTS

284 views • 27 slides

More recommend