hau at the germeval 2019 shared task on the
play

HAU at the GermEval 2019 Shared Task on the Identification of - PowerPoint PPT Presentation

Jul 14, 2023 •192 likes •493 views

HAU at the GermEval 2019 Shared Task on the Identification of Offensive Language in Microposts System Description of Word List, Statistical and Hybrid Approaches afer 1 , Tom De Smedt 2 , and Sylvia Jaki 3 Johannes Sch 1 Institute for

  1. HAU at the GermEval 2019 Shared Task on the Identification of Offensive Language in Microposts System Description of Word List, Statistical and Hybrid Approaches afer 1 , Tom De Smedt 2 , and Sylvia Jaki 3 Johannes Sch¨ 1 Institute for Information Science and Natural Language Processing, Hildesheim 2 Computational Linguistics Research Group, University of Antwerp 3 Department of Translation and Specialized Communication, U. of Hildesheim johannes.schaefer@uni-hildesheim.de , tom.desmedt@uantwerpen.be , jakisy@uni-hildesheim.de October 8th, 2019 Johannes, Tom and Sylvia HAU at GermEval 2019 October 8th, 2019 1 / 18

  2. Motivation Best performing systems from last year: Random forest ( ) and CNN ( ) GAP C C C C C O O O O O N N N N N V V V V V From our research: Manually created/annotated word list → combination possibilities? Johannes, Tom and Sylvia HAU at GermEval 2019 October 8th, 2019 2 / 18

  3. Overview POW Lexicon 4 1 Offensive Language Detection Systems 10 2 POW - HAU2 10 RF - HAU3 11 CNN - HAU1 12 Results, Conclusion and Outlook 16 3 Johannes, Tom and Sylvia HAU at GermEval 2019 October 8th, 2019 3 / 18

  4. POW Lexicon Overview POW Lexicon 4 1 Offensive Language Detection Systems 10 2 POW - HAU2 10 RF - HAU3 11 CNN - HAU1 12 Results, Conclusion and Outlook 16 3 Johannes, Tom and Sylvia HAU at GermEval 2019 October 8th, 2019 3 / 18

  5. POW Lexicon Overview POW List Profanity and Offensive Words (POW) Manually annotated dictionary which allows for the quantitative analysis of hate speech in a dataset Decision to work with a dictionary - result of GermEval 2018 List of 2852 words , mainly taken from German Twitter Embeddings (Ruppenhofer, 2018) Words either often used tendentiously in political contexts or vulgar/offensive Johannes, Tom and Sylvia HAU at GermEval 2019 October 8th, 2019 4 / 18

  6. POW Lexicon POW List: Types of Words Word classes (mostly) Nouns ( L¨ uge, Wesen, Arsch, Firlefanz ), incl. compounds ( Fremdenfeind, L¨ ugenpresse ) Also: adjectives ( bl¨ od, links-gr¨ un ) and participles ( verblendet ) Infinitives ( hetzen, spucken ) and imperatives ( lutsch, laber ) Interjections ( mimimi, boah ) Separate entries (tokens) Declensions ( Dreckschwein, Dreckschweine ) Conjugations ( labern, laber, labert ) Spelling variations ( schreien/schrein, scheiß/scheiss/scheis/chice ) Johannes, Tom and Sylvia HAU at GermEval 2019 October 8th, 2019 5 / 18

  7. POW Lexicon POW List: Annotation Annotation of intensity 0 tendentious os, AfDler, Staub, ¨ ( nichtmal, religi¨ Ubergriffe ) 1 tendentious, sensational ( heulen, unkontrolliert, Extremisten ) 2 demeaning ( Schnauze, stupide, Systemparteien, antideutsch ) 3 offensive (vulgar, racist) ( verbl¨ odet, Dreck, Honk, L¨ ugenpresse ) 4 offensive (extremely so) ( Hure, Untermenschen, Drecksau ) Johannes, Tom and Sylvia HAU at GermEval 2019 October 8th, 2019 6 / 18

  8. POW Lexicon POW List: Annotation of Types Johannes, Tom and Sylvia HAU at GermEval 2019 October 8th, 2019 7 / 18

  9. POW Lexicon POW List: Difficulties Context-dependence Intensity ( honk, verrecken, hurens¨ ohne ) Polarity ( bunt, willkommenskultur, fachkr¨ afte ) Type Lexial ambiguity ( geil, sack, fickt, w¨ urgen, schwuler, d¨ odel, muschi ) Grammatical ambiguity ( quatsch, blase, leeren, ritze ) ⇒ Pragmatic solution: Possibility for contextualisation by direct link to social media Johannes, Tom and Sylvia HAU at GermEval 2019 October 8th, 2019 8 / 18

  10. POW Lexicon POW List Johannes, Tom and Sylvia HAU at GermEval 2019 October 8th, 2019 9 / 18

  11. Offensive Language Detection Systems Overview POW Lexicon 4 1 Offensive Language Detection Systems 10 2 POW - HAU2 10 RF - HAU3 11 CNN - HAU1 12 Results, Conclusion and Outlook 16 3 Johannes, Tom and Sylvia HAU at GermEval 2019 October 8th, 2019 9 / 18

  12. Offensive Language Detection Systems POW - HAU2 Overview POW Lexicon 4 1 Offensive Language Detection Systems 10 2 POW - HAU2 10 RF - HAU3 11 CNN - HAU1 12 Results, Conclusion and Outlook 16 3 Johannes, Tom and Sylvia HAU at GermEval 2019 October 8th, 2019 9 / 18

  13. Offensive Language Detection Systems POW - HAU2 System HAU2: POW List Lookup → Tweet Motivation : Word lists are very explainable (cf. “black boxes”) and precise Method : For each message, check if it has words that are also in the POW list Compute the sum of the score of those words > threshold ⇒ offensive Mapping of intensity annotation (0-4 in POW list): 0 → 0.1, 1 → 0.25, 2 → 0.5, 3/4 → 1.0 For example : “Ungebildetes, kulturloses Gesindel f¨ uhrt Deutschland vor!” → ungebildet (0.5) + gesindel (1.0) = 1.5 > 0.95 ⇒ offensive Johannes, Tom and Sylvia HAU at GermEval 2019 October 8th, 2019 10 / 18

  14. Offensive Language Detection Systems POW - HAU2 System HAU2: POW List Lookup → Tweet Motivation : Word lists are very explainable (cf. “black boxes”) and precise Method : For each message, check if it has words that are also in the POW list Compute the sum of the score of those words > threshold ⇒ offensive Mapping of intensity annotation (0-4 in POW list): 0 → 0.1, 1 → 0.25, 2 → 0.5, 3/4 → 1.0 For example : “Ungebildetes, kulturloses Gesindel f¨ uhrt Deutschland vor!” → ungebildet (0.5) + gesindel (1.0) = 1.5 > 0.95 ⇒ offensive Results : Low recall for OFFENSE: 37.11% (lexicon should be expanded) Johannes, Tom and Sylvia HAU at GermEval 2019 October 8th, 2019 10 / 18

  15. Offensive Language Detection Systems RF - HAU3 Overview POW Lexicon 4 1 Offensive Language Detection Systems 10 2 POW - HAU2 10 RF - HAU3 11 CNN - HAU1 12 Results, Conclusion and Outlook 16 3 Johannes, Tom and Sylvia HAU at GermEval 2019 October 8th, 2019 10 / 18

  16. Offensive Language Detection Systems RF - HAU3 System HAU3: Random Forest Motivation : among last year’s best systems, use as comparative baseline Python algorithm : https://github.com/textgain/grasp Features : character trigrams + word unigrams 100 trees, each with a random subset of 750 features Johannes, Tom and Sylvia HAU at GermEval 2019 October 8th, 2019 11 / 18

  17. Offensive Language Detection Systems CNN - HAU1 Overview POW Lexicon 4 1 Offensive Language Detection Systems 10 2 POW - HAU2 10 RF - HAU3 11 CNN - HAU1 12 Results, Conclusion and Outlook 16 3 Johannes, Tom and Sylvia HAU at GermEval 2019 October 8th, 2019 11 / 18

  18. Offensive Language Detection Systems CNN - HAU1 Starting Point: NN Architecture Sch¨ afer (2018) at GermEval 2018; extended from Founta et al. (2018) Text Input (Tweet) Part-of-Speech tags Metadata Text Encoder Meta Encoder Encoder (LSTM) (Densely-connected NN) (LSTM/Dense) concatenate ˆ y Johannes, Tom and Sylvia HAU at GermEval 2019 October 8th, 2019 12 / 18

  19. Offensive Language Detection Systems CNN - HAU1 Our Basic NN Architecture for GermEval 2019 Text Input (Tweet) Metadata Text Encoder Meta Encoder (CNN 1 ) (Densely-connected NN) concatenate y ˆ 1 CNN configuration as described in Sch¨ afer and Burtenshaw (2019) Johannes, Tom and Sylvia HAU at GermEval 2019 October 8th, 2019 13 / 18

  20. Offensive Language Detection Systems CNN - HAU1 Our Basic NN Architecture for GermEval 2019 Text Input (Tweet) Metadata Text Encoder Meta Encoder (CNN 1 ) (Densely-connected NN) concatenate y ˆ ML improvements: early stopping; class weights 1 CNN configuration as described in Sch¨ afer and Burtenshaw (2019) Johannes, Tom and Sylvia HAU at GermEval 2019 October 8th, 2019 13 / 18

  21. Offensive Language Detection Systems CNN - HAU1 Our Basic NN Architecture for GermEval 2019 Text Input (Tweet) Metadata Text Encoder Meta Encoder (CNN 1 ) (Densely-connected NN) concatenate y ˆ ML improvements: early stopping; class weights → POW list features? 1 CNN configuration as described in Sch¨ afer and Burtenshaw (2019) Johannes, Tom and Sylvia HAU at GermEval 2019 October 8th, 2019 13 / 18

  22. Offensive Language Detection Systems CNN - HAU1 HAU1: CNN + POW List Model Johannes, Tom and Sylvia HAU at GermEval 2019 October 8th, 2019 14 / 18

  23. Offensive Language Detection Systems CNN - HAU1 Results on the GermEval Training Dataset Average scores from 3-fold cross validation (values in %): System configuration Accuracy F 1 -score OTHER OFFENSE m.-avg. CNN 76.25 83.02 60.47 71.98 CNN + meta 76.10 82.23 63.43 72.84 CNN + meta POW 78.15 83.77 66.56 75.17 CNN POW + meta 76.67 82.62 64.45 73.56 CNN POW + meta POW 78.87 84.62 66.21 75.46 Johannes, Tom and Sylvia HAU at GermEval 2019 October 8th, 2019 15 / 18

  24. Results, Conclusion and Outlook Overview POW Lexicon 4 1 Offensive Language Detection Systems 10 2 POW - HAU2 10 RF - HAU3 11 CNN - HAU1 12 Results, Conclusion and Outlook 16 3 Johannes, Tom and Sylvia HAU at GermEval 2019 October 8th, 2019 15 / 18

  25. Results, Conclusion and Outlook Overview System Runs HAU1-3 for Tasks 1-3 F 1 -scores on the GermEval 2019 test dataset Subtask I (OL detection): HAU2 (POW list lookup) 68.13% HAU3 (random forest) 69.75% HAU1 (CNN+meta including POW) 70.46% Subtask II (fine-grained OL detection): HAU3 (random forest) 40.80% HAU1 (CNN+meta including POW) 45.34% Subtask III (implicit/explicit): HAU1 (CNN+meta including POW) 69.3% Johannes, Tom and Sylvia HAU at GermEval 2019 October 8th, 2019 16 / 18

Download Presentation
Download Policy: The content available on the website is offered to you 'AS IS' for your personal information and use only. It cannot be commercialized, licensed, or distributed on other websites without prior consent from the author. To download a presentation, simply click this link. If you encounter any difficulties during the download process, it's possible that the publisher has removed the file from their server.

Recommend

LAW-MWE-CxG 2018 Shared task poster boosters 1. DEEP-BGT AT PARSEME SHARED TASK 2018:

LAW-MWE-CxG 2018 Shared task poster boosters 1. DEEP-BGT AT PARSEME SHARED TASK 2018:

LAW-MWE-CxG 2018 Shared task poster boosters 1. DEEP-BGT AT PARSEME SHARED TASK 2018: BIDIRECTIONAL LSTM- CRF MODEL FOR VERBAL MULTIWORD EXPRESSION IDENTIFICATION, Gzde Berk, Berna Erden and Tunga Gungor 2. GBD-NER AT PARSEME SHARED TASK

555 views • 14 slides
TUPA at MRP 2019 A Multi-Task Baseline System CoNLL Shared Task 3 November 2019 1 / 9 Daniel

TUPA at MRP 2019 A Multi-Task Baseline System CoNLL Shared Task 3 November 2019 1 / 9 Daniel

TUPA at MRP 2019 A Multi-Task Baseline System CoNLL Shared Task 3 November 2019 1 / 9 Daniel Hershcovich 1 , Ofjr Arviv 2 1 University of Copenhagen 2 Hebrew University of Jerusalem Transition-Based UCCA Parser look (Hershcovich et al., 2017)

405 views • 12 slides
Shared Governance Task Force Report https://web.ramapo.edu/shared-governance-task-force/ 1

Shared Governance Task Force Report https://web.ramapo.edu/shared-governance-task-force/ 1

4/4/2018 Ramapo College of New Jersey Shared Governance Task Force Report https://web.ramapo.edu/shared-governance-task-force/ 1 Overview of Preliminary Report How Did We Get Here? The Charge to the Shared Governance Task Force (SGTF)

228 views • 9 slides
The PARSEME Shared Task on Automatic Identification of Verbal Multiword Expressions (1.1) C.

The PARSEME Shared Task on Automatic Identification of Verbal Multiword Expressions (1.1) C.

Context Annotation methodology Corpora Shared task Wrapping up The PARSEME Shared Task on Automatic Identification of Verbal Multiword Expressions (1.1) C. Ramisch 1 , S. Ricardo 1 , A. Savary 2 , V. Vincze 3 , V. Barbu 4 , A. Bhatia 5 , M.

1.02k views • 25 slides
The SIGMORPHON 2016 shared task morphological reinflection Ryan Cotterell, Christo Kirov,

The SIGMORPHON 2016 shared task morphological reinflection Ryan Cotterell, Christo Kirov,

The SIGMORPHON 2016 shared task morphological reinflection Ryan Cotterell, Christo Kirov, John Sylak-Glassman, David Yarowsky, Jason Eisner, Mans Hulden 1 SIGMORPHON 2016 Shared task - morphological reinflection Shared task

1.28k views • 73 slides
WMT 10 Shared Tasks: Translation Task System Combination Task Chris Callison-Burch, Philipp

WMT 10 Shared Tasks: Translation Task System Combination Task Chris Callison-Burch, Philipp

WMT 10 Shared Tasks: Translation Task System Combination Task Chris Callison-Burch, Philipp Koehn, Christof Monz, Omar Zaidan 15 July 2010 Philipp Koehn WMT10 Shared Tasks 15 July 2010 Translation Task 1 Open benchmark for machine

662 views • 37 slides
VLSP 2019 Shared Task: Dependency Parsing NGUYEN Thi Minh Huyen Hanoi - 2019 Outline

VLSP 2019 Shared Task: Dependency Parsing NGUYEN Thi Minh Huyen Hanoi - 2019 Outline

VLSP 2019 Shared Task: Dependency Parsing NGUYEN Thi Minh Huyen Hanoi - 2019 Outline Introduction 1 Data Preparation 2 Evaluation 3 Results 4 Award Presentation 5 NGUYEN Thi Minh Huyen Dependency Parsing - VLSP 2019 1 / 28

529 views • 29 slides
Modeling Unrestricted Coreference in OntoNotes CoNLL-2011 Shared Task Sameer S Pradhan 1 Lance

Modeling Unrestricted Coreference in OntoNotes CoNLL-2011 Shared Task Sameer S Pradhan 1 Lance

CoNLL Shared Task OntoNotes Evaluation Modeling Unrestricted Coreference in OntoNotes CoNLL-2011 Shared Task Sameer S Pradhan 1 Lance Ramshaw 1 Mitchell Marcus 2 Martha Palmer 3 Ralph Weischedel 1 Nianwen Xue 4 1 BBN Technologies, Cambridge, MA

1.17k views • 70 slides
Results of the WMT19 Metrics Shared Task Segment-Level and Strong MT Systems Pose Big Challenges

Results of the WMT19 Metrics Shared Task Segment-Level and Strong MT Systems Pose Big Challenges

Results of the WMT19 Metrics Shared Task Segment-Level and Strong MT Systems Pose Big Challenges Qingsong Ma Johnny Tian-Zheng Wei Ond rej Bojar Yvette Graham 1 / 31 Overview Overview of Metrics Task Updates to Metric Task in

554 views • 40 slides
author profiling shared task on: Bots and gender profiling Francisco Rangel & Paolo Rosso

author profiling shared task on: Bots and gender profiling Francisco Rangel & Paolo Rosso

Overview of the 7 th author profiling shared task on: Bots and gender profiling Francisco Rangel & Paolo Rosso 10th September 2019 Bots: propaganda, fake news, inflammatory content Bots may influence users with comercial, political or

512 views • 35 slides
WASC 2019 Findings Presentation to stakeholders February 2019 2 3 Process Task 1 Task 2

WASC 2019 Findings Presentation to stakeholders February 2019 2 3 Process Task 1 Task 2

WASC 2019 Findings Presentation to stakeholders February 2019 2 3 Process Task 1 Task 2 Update the Task 3 Update and progress report Assess the analyze our on our 2013 school program school profile action alan Task 4 Task 5

373 views • 15 slides
TRAVERSAL at PARSEME Shared Task 2018: Identification of VMWEs Using a Discriminative

TRAVERSAL at PARSEME Shared Task 2018: Identification of VMWEs Using a Discriminative

TRAVERSAL at PARSEME Shared Task 2018: Identification of VMWEs Using a Discriminative Tree-Structured Model Jakub Waszczuk Heinrich Heine University, Dsseldorf, Germany August 25, 2018 1 / 7 VMWEs, (dis)continuity, and sequential models

468 views • 12 slides
FEVER shared Task Tariq Alhindi 08/22/2018 Motivation 67% of consumers now look online

FEVER shared Task Tariq Alhindi 08/22/2018 Motivation 67% of consumers now look online

FEVER shared Task Tariq Alhindi 08/22/2018 Motivation 67% of consumers now look online for information before heading to a physical shop Yet, 61% of independent businesses, including restaurants, hairdressers, pharmacists and

523 views • 19 slides
Results of the WMT16 Metrics Shared Task Ond rej Bojar Yvette Graham Amir Kamran Milo s

Results of the WMT16 Metrics Shared Task Ond rej Bojar Yvette Graham Amir Kamran Milo s

Results of the WMT16 Metrics Shared Task Ond rej Bojar Yvette Graham Amir Kamran Milo s Stanojevi c WMT16, Aug 11, 2016 1 / 32 Overview Summary of Metrics Task. Updates to Metric Task in 2016. Results 2 / 32 Metrics

634 views • 45 slides
MI MI and Shared MI MI and Shared and Shared Decision Making and Shared Decision Making

MI MI and Shared MI MI and Shared and Shared Decision Making and Shared Decision Making

The project PRECIOUS has received funding from the European Union s Seventh Framework Programme under Grant Agreement n 611366" MI MI and Shared MI MI and Shared and Shared Decision Making and Shared Decision Making Decision Making

412 views • 20 slides
T RA P ACC and T RA P ACC S at PARSEME Shared Task 2018: Neural Transition Tagging of Verbal

T RA P ACC and T RA P ACC S at PARSEME Shared Task 2018: Neural Transition Tagging of Verbal

T RA P ACC and T RA P ACC S at PARSEME Shared Task 2018: Neural Transition Tagging of Verbal Multiword Expressions Regina Stodden, Behrang Qasemizadeh, Laura Kallmeyer regina.stodden@uni-duesseldorf.de SFB991, Heinrich Heine University D

224 views • 5 slides
Exploring the Design Space for Adaptive Graphical User Interfaces Krzysztof Gajos (University of

Exploring the Design Space for Adaptive Graphical User Interfaces Krzysztof Gajos (University of

Exploring the Design Space for Adaptive Graphical User Interfaces Krzysztof Gajos (University of Washington) Mary Czerwinski (Microsoft Research) Desney Tan (Microsoft Research) Daniel S. Weld (University of Washington) Scope Graphical

1.43k views • 105 slides
1 IP Multicast Applications IP Multicast Applications Agenda Agenda Fundamental Approaches:

1 IP Multicast Applications IP Multicast Applications Agenda Agenda Fundamental Approaches:

Basic Motivations Basic Motivations Mobile Network

683 views • 9 slides
Annotation of Tense & Aspect Semantics for Sentential AMR Lucia Donatelli 1 , Michael Regan 2

Annotation of Tense & Aspect Semantics for Sentential AMR Lucia Donatelli 1 , Michael Regan 2

Annotation of Tense & Aspect Semantics for Sentential AMR Lucia Donatelli 1 , Michael Regan 2 , William Croft 2 , & Nathan Schneider 1 Georgetown University 1 , University of New Mexico 2 LAW-MWE-CxG 2018 26 August 2018 Why tense and

849 views • 59 slides
Evaluating Dialogue Act Tagging with Naive & Expert annotators Jeroen Geertzen & Volha

Evaluating Dialogue Act Tagging with Naive & Expert annotators Jeroen Geertzen & Volha

Tilburg University Evaluating Dialogue Act Tagging with Naive & Expert annotators Jeroen Geertzen & Volha Petukhova & Harry Bunt LREC 2008 / Marrakech / May 28 th Jeroen Geertzen & Volha Petukhova & Harry Bunt: Evaluating

383 views • 25 slides
How to play games with types Robin Cooper Centre for Linguistic Theory and Studies in Probability

How to play games with types Robin Cooper Centre for Linguistic Theory and Studies in Probability

How to play games with types Robin Cooper Centre for Linguistic Theory and Studies in Probability (CLASP) Department of Philosophy, Linguistics and Theory of Science (FLoV) Supported in part by VR project 2016-01162, Incremental Reasoning in

790 views • 47 slides
About the Optimal Optical Transfer Function of a Microscope Jrg Enderlein Georg August

About the Optimal Optical Transfer Function of a Microscope Jrg Enderlein Georg August

About the Optimal Optical Transfer Function of a Microscope Jrg Enderlein Georg August University Gttingen Germany Quantitative BioImaging 2019, January 9 11 Rennes, France The Optical Microscope And Its Resolution

446 views • 30 slides
Metaphor and its Computational Processing Alexis Palmer (slides from Michaela Regneri)

Metaphor and its Computational Processing Alexis Palmer (slides from Michaela Regneri)

Metaphor and its Computational Processing Alexis Palmer (slides from Michaela Regneri) Einfhrung in der Pragmatik & Diskurs July 7, 2014 Monday, July 7, 14 Metaphors Brad Pitt ist ein Chamleon. Metaphor & Processing Metaphor

796 views • 62 slides
What Syntax Feeds Semantics? ESSLLI 2008 Workshop Hamburg, 11-15 August, 2008 Maribel Romero

What Syntax Feeds Semantics? ESSLLI 2008 Workshop Hamburg, 11-15 August, 2008 Maribel Romero

What Syntax Feeds Semantics? ESSLLI 2008 Workshop Hamburg, 11-15 August, 2008 Maribel Romero University of Konstanz 1 Division of labor between Syntax and Semantics Frege s Principle of Compositionality: The meaning of a complex

436 views • 29 slides

More recommend