From Sentiment to Emotion: Challenges of a More Fine-Grained - - PowerPoint PPT Presentation

▶
from sentiment to emotion
SMART_READER_LITE
LIVE PREVIEW

From Sentiment to Emotion: Challenges of a More Fine-Grained - - PowerPoint PPT Presentation

Jan 23, 2024 339 likes •1.4k views

Invited Talk at IMS, Universitt Stuttgart Stuttgart, November 26, 2018 From Sentiment to Emotion: Challenges of a More Fine-Grained Analysis of Affective Language Sven Buechel Jena University Language and Information Engineering (JULIE) Lab

slide-1
SLIDE 1

Invited Talk at IMS, Universität Stuttgart Stuttgart, November 26, 2018 Sven Buechel From Sentiment to Emotion

1

From Sentiment to Emotion:

Challenges of a More Fine-Grained Analysis of Affective Language

Sven Buechel

Slides: https://julielab.de/downloads/publications/slides/buechel_invited_ims_2018.pdf

Jena University Language and Information Engineering (JULIE) Lab Friedrich-Schiller-Universität Jena, Jena, Germany https://julielab.de

slide-2
SLIDE 2

Invited Talk at IMS, Universität Stuttgart Stuttgart, November 26, 2018 Sven Buechel From Sentiment to Emotion

2

Outline

  • Introduction
  • Applications of emotion analysis in DH and CSS
  • Dealing with lack of interoperability
  • Dealing with data sparsity
  • Discussion and conclusion
slide-3
SLIDE 3

Invited Talk at IMS, Universität Stuttgart Stuttgart, November 26, 2018 Sven Buechel From Sentiment to Emotion

3

Outline

Ø Introduction

  • Applications of emotion analysis in DH and CSS
  • Dealing with lack of interoperability
  • Dealing with data sparsity
  • Discussion and conclusion
slide-4
SLIDE 4

Invited Talk at IMS, Universität Stuttgart Stuttgart, November 26, 2018 Sven Buechel From Sentiment to Emotion

4

Sentiment Analysis — Two-Class Problem

+ –

sunshine

I hate John Doe, he has a terrible sense of humor.

Ma Macbeth

slide-5
SLIDE 5

Invited Talk at IMS, Universität Stuttgart Stuttgart, November 26, 2018 Sven Buechel From Sentiment to Emotion

5

Sentiment Analysis — Multi-Class Problem

++ – – + –

sunshine

I hate John Doe, he has a terrible sense of humor.

Ma Macbeth

slide-6
SLIDE 6

Invited Talk at IMS, Universität Stuttgart Stuttgart, November 26, 2018 Sven Buechel From Sentiment to Emotion

6

Emotion Analysis

sunshine

I hate John Doe, he has a terrible sense of humor.

Ma Macbeth

slide-7
SLIDE 7

Invited Talk at IMS, Universität Stuttgart Stuttgart, November 26, 2018 Sven Buechel From Sentiment to Emotion

7

Emotion Analysis

???

sunshine

I hate John Doe, he has a terrible sense of humor.

Ma Macbeth

slide-8
SLIDE 8

Invited Talk at IMS, Universität Stuttgart Stuttgart, November 26, 2018 Sven Buechel From Sentiment to Emotion

8

Major Approaches in Emotion Representation

Prediction Problem Psychology Model discrete

emotions as instances

dimensional

emotions as compositions

Classification Regression

slide-9
SLIDE 9

Invited Talk at IMS, Universität Stuttgart Stuttgart, November 26, 2018 Sven Buechel From Sentiment to Emotion

9

Ekman’s Basic Emotions

Source: http://ocw.mit.edu/courses/brain-and-cognitive-sciences/9-00sc-introduction-to-psychology-fall-2011/emotion-motivation/discussion-emotion/

slide-10
SLIDE 10

Invited Talk at IMS, Universität Stuttgart Stuttgart, November 26, 2018 Sven Buechel From Sentiment to Emotion

10

Representing Emotion — Wheel of Emotion

Source: https://en.wikipedia.org/wiki/Contrasting_and_categorization_of_emotions#/media/File:Plutchik-wheel.svg

slide-11
SLIDE 11

Invited Talk at IMS, Universität Stuttgart Stuttgart, November 26, 2018 Sven Buechel From Sentiment to Emotion

11

Major Approaches in Emotion Representation

0.1 0.2 … 0.9

Psychology Model discrete

emotions as instances

dimensional

emotions as compositions

Prediction Problem Classification Regression

slide-12
SLIDE 12

Invited Talk at IMS, Universität Stuttgart Stuttgart, November 26, 2018 Sven Buechel From Sentiment to Emotion

12

Valence-Arousal-Dominance

−1.0 −0.5 0.0 0.5 1.0 −1.0 −0.5 0.0 0.5 1.0 −1.0 −0.5 0.0 0.5 1.0

  • Anger

Surprise Disgust Fear Sadness Joy

Valence

(displeasure—pleasure)

Arousal

( c a l m n e s s — e x c i t e m e n t )

Dominance

(being controlled—in control) (Russell & Mehrabian, 1977)

slide-13
SLIDE 13

Invited Talk at IMS, Universität Stuttgart Stuttgart, November 26, 2018 Sven Buechel From Sentiment to Emotion

13

Valence-Arousal

(Russell, 2003) high arousal low arousal high valence low valence

happy excited tense upset sad tired calm serene

slide-14
SLIDE 14

Invited Talk at IMS, Universität Stuttgart Stuttgart, November 26, 2018 Sven Buechel From Sentiment to Emotion

14

Major Approaches in Emotion Representation

0.1 0.2 … 0.9

Psychology Model discrete

emotions as instances

dimensional

emotions as compositions

Prediction Problem

+Valence +Arousal

  • Dominance

0.8 Valence 0.6 Arousal

  • 0.3 Dominance

Classification Regression

slide-15
SLIDE 15

Invited Talk at IMS, Universität Stuttgart Stuttgart, November 26, 2018 Sven Buechel From Sentiment to Emotion

15

Major Approaches in Emotion Representation

Classification Regression Psychology Model discrete

emotions as instances

dimensional

emotions as compositions

Prediction Problem

+Valence +Arousal

  • Dominance

0.8 Valence 0.6 Arousal

  • 0.3 Dominance

0.1 0.2 … 0.9

slide-16
SLIDE 16

Invited Talk at IMS, Universität Stuttgart Stuttgart, November 26, 2018 Sven Buechel From Sentiment to Emotion

16

Current Situation in Emotion Analysis

  • Huge interest
  • Very messy

– lack of agreed upon terminology – no consensus w.r.t. emotion representation

  • Consequences

– data sparsity – lack of interoperability of datasets, tools and analyses

  • But getting better

– shared tasks (SemEval 2018, 2019; WASSA 2017, 2018) – growing awareness of psychological work – work specifically aiming at enhancing interoperability e.g., Bostan & Klinger (COLING 2018); our own work

slide-17
SLIDE 17

Invited Talk at IMS, Universität Stuttgart Stuttgart, November 26, 2018 Sven Buechel From Sentiment to Emotion

17

Outline

Ø Introduction

  • Applications of emotion analysis in DH and CSS
  • Dealing with lack of interoperability
  • Dealing with data sparsity
  • Discussion and conclusion
slide-18
SLIDE 18

Invited Talk at IMS, Universität Stuttgart Stuttgart, November 26, 2018 Sven Buechel From Sentiment to Emotion

18

Outline

  • Introduction

Ø Applications of emotion analysis in DH and CSS

  • Dealing with lack of interoperability
  • Dealing with data sparsity
  • Discussion and conclusion
slide-19
SLIDE 19

Invited Talk at IMS, Universität Stuttgart Stuttgart, November 26, 2018 Sven Buechel From Sentiment to Emotion

19

Measuring Organizational Emotion

  • Collaboration with management and organization researchers
  • Interest in anthropomorphic communication behavior of
  • rganizations (esp. targets, virtues, cognitive processes)
  • Is this framework also applicable to emotions?
  • Do enterprises communicate with a distinctive and persistent

emotional profile?

  • Analysis of annual reports and corporate social responsibility

(CSR) reports

(Buechel et al., WASSA 2016)

slide-20
SLIDE 20

Invited Talk at IMS, Universität Stuttgart Stuttgart, November 26, 2018 Sven Buechel From Sentiment to Emotion

20

Annual Reports

The updated Mercedes-Benz Sprinter appeals with a new design and a world first. The Sprinter is the first van series worldwide for which all models can be supplied with the ESP electronic stability program.

The most important new model in 2002 was the Actros, which had its premiere at the International Auto Show (IAA) in Hanover and was well received by customers and automotive journalists. Its distinctive characteristics are its more powerful engines, a new axle and suspension concept, improved aerodynamics and a redesigned driver’s cab. Mercedes-Benz Vans still leads the field The Mercedes-Benz Vans business unit sold 236,600 vehicles worldwide in 2002, nearly matching the figure for 2001. With a market share of 18% (2001: 19%) in the segment of 2 to 6 metric tons, Mercedes-Benz Vans is still the market leader in Western Europe. Whereas the Sprinter was able to maintain its strong market position in the heavy vans segment, in the segment of mid-size vans the market share of the Vito decreased due to the model changeover scheduled for 2003. In the spring of 2002, DaimlerChrysler introduced the new Vaneo, which is positioned as a premium product in this segment. The updated Sprinter model was introduced at the International Auto Show (IAA) in Hanover in September

  • 2002. This new model is more attractive and, thanks

to longer service intervals, more economical. Another new feature is the Electronic Stability Program (ESP). DaimlerChrysler is the first vehicle manufacturer to offer this system in this van segment. To strengthen its presence in the US van market in early 2003, Daimler- Chrysler plans to offer the Sprinter, which has been sold successfully in the US under the Freightliner brand name since the middle of 2001, as a Dodge brand vehicle as well. We also plan to launch the Sprinter in Canada and Mexico. The licensing agreement with Volkswagen AG for the production of the Sprinter van by Volkswagen was renewed to cover successor models as well.

02/01

in %

1,000

Units

Unit Sales 2002 1

1 Wholesale figures (including leased vehicles) 2 Including the Mitsubishi L200 pickup and the Mitsubishi Pajero in South Africa 3 Including schoolbuses by Thomas Built Buses and bus chassis by Freightliner

World

  • f which:Vans 2

Trucks 3 Buses Unimogs Europe

  • f which:Germany

Western Europe (excluding Germany)

  • f which:France

United Kingdom Italy NAFTA

  • f which:United States

South America (excluding Mexico)

  • f which:Brazil

Asia 485 246 212 25 2 287 103 162 32 33 23 1 18 100 37 30 24

  • 2
  • 5

+ 3

  • 8
  • 23
  • 2
  • 3
  • 5
  • 10

+ 14 + 4 + 1 1 + 12

  • 14
  • 12
  • 8

DaimlerChrysler, 2002

slide-21
SLIDE 21

Invited Talk at IMS, Universität Stuttgart Stuttgart, November 26, 2018 Sven Buechel From Sentiment to Emotion

21

Annual Reports

The updated Mercedes-Benz Sprinter appeals with a new design and a world first. The Sprinter is the first van series worldwide for which all models can be supplied with the ESP electronic stability program.

The most important new model in 2002 was the Actros, which had its premiere at the International Auto Show (IAA) in Hanover and was well received by customers and automotive journalists. Its distinctive characteristics are its more powerful engines, a new axle and suspension concept, improved aerodynamics and a redesigned driver’s cab. Mercedes-Benz Vans still leads the field The Mercedes-Benz Vans business unit sold 236,600 vehicles worldwide in 2002, nearly matching the figure for 2001. With a market share of 18% (2001: 19%) in the segment of 2 to 6 metric tons, Mercedes-Benz Vans is still the market leader in Western Europe. Whereas the Sprinter was able to maintain its strong market position in the heavy vans segment, in the segment of mid-size vans the market share of the Vito decreased due to the model changeover scheduled for 2003. In the spring of 2002, DaimlerChrysler introduced the new Vaneo, which is positioned as a premium product in this segment. The updated Sprinter model was introduced at the International Auto Show (IAA) in Hanover in September

  • 2002. This new model is more attractive and, thanks

to longer service intervals, more economical. Another new feature is the Electronic Stability Program (ESP). DaimlerChrysler is the first vehicle manufacturer to offer this system in this van segment. To strengthen its presence in the US van market in early 2003, Daimler- Chrysler plans to offer the Sprinter, which has been sold successfully in the US under the Freightliner brand name since the middle of 2001, as a Dodge brand vehicle as well. We also plan to launch the Sprinter in Canada and Mexico. The licensing agreement with Volkswagen AG for the production of the Sprinter van by Volkswagen was renewed to cover successor models as well.

02/01

in %

1,000

Units

Unit Sales 2002 1

1 Wholesale figures (including leased vehicles) 2 Including the Mitsubishi L200 pickup and the Mitsubishi Pajero in South Africa 3 Including schoolbuses by Thomas Built Buses and bus chassis by Freightliner

World

  • f which:Vans 2

Trucks 3 Buses Unimogs Europe

  • f which:Germany

Western Europe (excluding Germany)

  • f which:France

United Kingdom Italy NAFTA

  • f which:United States

South America (excluding Mexico)

  • f which:Brazil

Asia 485 246 212 25 2 287 103 162 32 33 23 1 18 100 37 30 24

  • 2
  • 5

+ 3

  • 8
  • 23
  • 2
  • 3
  • 5
  • 10

+ 14 + 4 + 1 1 + 12

  • 14
  • 12
  • 8

DaimlerChrysler, 2002

Mercedes-Benz Vans still leads the field The Mercedes-Benz Vans business unit sold 236,600 vehicles worldwide in 2002, nearly matching the figure for 2001. With a market share of 18% (2001: 19%) in the segment of 2 to 6 metric tons, Mercedes-Benz Vans is still the market leader in Western Europe. Whereas the

slide-22
SLIDE 22

Invited Talk at IMS, Universität Stuttgart Stuttgart, November 26, 2018 Sven Buechel From Sentiment to Emotion

22

Corporate Social Responsibility (CSR) Reports

McDonald‘s 2012/13

slide-23
SLIDE 23

Invited Talk at IMS, Universität Stuttgart Stuttgart, November 26, 2018 Sven Buechel From Sentiment to Emotion

23

Choosing an Emotion Representation

  • Most of the documents are rather neutral

– fine-grained, „high-resolution“

  • Exploratory study

– unclear what emotion categories are most relevant

  • Social science application

– interpretable outcome

−1.0 −0.5 0.0 0.5 1.0 −1.0 −0.5 0.0 0.5 1.0 −1.0 −0.5 0.0 0.5 1.0

  • Anger

Surprise Disgust Fear Sadness Joy

slide-24
SLIDE 24

Invited Talk at IMS, Universität Stuttgart Stuttgart, November 26, 2018 Sven Buechel From Sentiment to Emotion

24

Corpus Description

  • Countries: US, UK, Germany
  • 30 companies per country

(DIJA, FTSE 100, DAX)

  • 1676 documents (2/3 AR, 1/3 CSR)
  • Years 1992–2015
  • Successor: JOCo (Händschke et al., ECONLP @ ACL 2018)
slide-25
SLIDE 25

Invited Talk at IMS, Universität Stuttgart Stuttgart, November 26, 2018 Sven Buechel From Sentiment to Emotion

25

Measuring Document Emotion: JEMAS

Available:

https://github.com/JULIELab/JEmAS

(Buechel & Hahn, ECAI 2016)

(sunshine, <8, 3, 5>) (terrorism, <2, 7, 3>) (calm, <7, 2, 7>)

full text documents linguistic normalization BOW representation

emotion lexicon

VAD score calculation

slide-26
SLIDE 26

Invited Talk at IMS, Universität Stuttgart Stuttgart, November 26, 2018 Sven Buechel From Sentiment to Emotion

26

Results — Annual vs. CSR Reports

0.3 0.5 0.7 0.9 −1.20 −1.15 −1.10 −1.05 −1.00 −0.95 −0.90 Valence Arousal

ANN CSR

0.4 0.6 0.8 1.0 0.4 0.5 0.6 0.7 0.8 0.9 Valence Dominance −1.20 −1.05 −0.90 0.4 0.5 0.6 0.7 0.8 0.9 Arousal Dominance

(Buechel et al., WASSA 2016)

slide-27
SLIDE 27

Invited Talk at IMS, Universität Stuttgart Stuttgart, November 26, 2018 Sven Buechel From Sentiment to Emotion

27

Results — Emotional Profiling of Organizations

0.50 0.60 0.70 0.80 −1.05 −1.00 −0.95 Valence Arousal

Microsoft Deutsche_Bank MerckCo Siemens Daimler Lufthansa BMW Intel

0.50 0.60 0.70 0.80 0.55 0.60 0.65 0.70 Valence Dominance

Microsoft Deutsche_Bank MerckCo Siemens Daimler Lufthansa BMW Intel

  • Statistical analysis revealed that…

– authoring company explains most of variability in VAD score – VAD scores are rather time invariant

  • Companies have distinct and persistent emotional profile
slide-28
SLIDE 28

Invited Talk at IMS, Universität Stuttgart Stuttgart, November 26, 2018 Sven Buechel From Sentiment to Emotion

28

DH Application: Emotional Profiling in the DTA

Source and License: Charles Hackley via https://flic.kr/p/qSsjHA (CC-BY 2.0)

slide-29
SLIDE 29

Invited Talk at IMS, Universität Stuttgart Stuttgart, November 26, 2018 Sven Buechel From Sentiment to Emotion

29

Emotional Profiles of Literary Forms in the DTA

−8 −4 2 4 −4 −2 2 Valence Arousal

Lyric Narratives Drama

−4 −2 2 −4 2 4 Arousal Dominance −8 −4 2 4 −4 2 4 Valence Dominance

(Buechel et al., LT4DH 2016, DH 2017)

slide-30
SLIDE 30

Invited Talk at IMS, Universität Stuttgart Stuttgart, November 26, 2018 Sven Buechel From Sentiment to Emotion

30

Exploring Historical Word Emotions: heart

JeSemE.org

(Hellrich et al., COLING 2018)

slide-31
SLIDE 31

Invited Talk at IMS, Universität Stuttgart Stuttgart, November 26, 2018 Sven Buechel From Sentiment to Emotion

31

Interim Conclusion

  • Great potential of emotion analysis for DH and CSS
  • Fine-grained representations more informative than polarity
  • Quite simple methodologies
slide-32
SLIDE 32

Invited Talk at IMS, Universität Stuttgart Stuttgart, November 26, 2018 Sven Buechel From Sentiment to Emotion

32

Outline

  • Introduction

Ø Applications of emotion analysis in DH and CSS

  • Dealing with lack of interoperability
  • Dealing with data sparsity
  • Discussion and conclusion
slide-33
SLIDE 33

Invited Talk at IMS, Universität Stuttgart Stuttgart, November 26, 2018 Sven Buechel From Sentiment to Emotion

33

Outline

  • Introduction
  • Applications of emotion analysis in DH and CSS

Ø Dealing with lack of interoperability

  • Dealing with data sparsity
  • Discussion and conclusion
slide-34
SLIDE 34

Invited Talk at IMS, Universität Stuttgart Stuttgart, November 26, 2018 Sven Buechel From Sentiment to Emotion

34

Emotion Representation Mapping

  • How to compare JEmAS against previous work?
  • Basic idea: find a mapping that converts VAD to BE scores
  • Also interesting for psych. theory: what is the relationship

between discrete and dimensional emotion representations?

  • Psychologist already created double annotated lexicons for

this reason!

−1.0 −0.5 0.0 0.5 1.0 −1.0 −0.5 0.0 0.5 1.0 −1.0 −0.5 0.0 0.5 1.0

  • Anger

Surprise Disgust Fear Sadness Joy

slide-35
SLIDE 35

Invited Talk at IMS, Universität Stuttgart Stuttgart, November 26, 2018 Sven Buechel From Sentiment to Emotion

35

Emotion Representation Mapping

Word Val Aro Dom Joy Ang Sadn Fear Disg sunshine 7.3 3.2 6.8 4.1 1.3 1.2 1.1 1.3 terrorism 1.8 8.1 4.2 1.4 4.1 3.2 3.8 3.6 earthquake 1.9 8.3 1.9 1.2 3.2 3.8 4.3 2.7

slide-36
SLIDE 36

Invited Talk at IMS, Universität Stuttgart Stuttgart, November 26, 2018 Sven Buechel From Sentiment to Emotion

36

Emotion Representation Mapping

Word Val Aro Dom Joy Ang Sadn Fear Disg sunshine 7.3 3.2 6.8 4.1 1.3 1.2 1.1 1.3 terrorism 1.8 8.1 4.2 1.4 4.1 3.2 3.8 3.6 earthquake 1.9 8.3 1.9 1.2 3.2 3.8 4.3 2.7 ML features prediction ML features prediction

(Buechel & Hahn, ECAI 2016)

slide-37
SLIDE 37

Invited Talk at IMS, Universität Stuttgart Stuttgart, November 26, 2018 Sven Buechel From Sentiment to Emotion

37

Emotion Representation Mapping

Word Val Aro Dom Joy Ang Sadn Fear Disg sunshine 7.3 3.2 6.8 4.1 1.3 1.2 1.1 1.3 terrorism 1.8 8.1 4.2 1.4 4.1 3.2 3.8 3.6 earthquake 1.9 8.3 1.9 1.2 3.2 3.8 4.3 2.7

Map JEmAS output to BE — SOTA in three emotion categories!

ML features prediction ML features prediction

(Buechel & Hahn, ECAI 2016)

slide-38
SLIDE 38

Invited Talk at IMS, Universität Stuttgart Stuttgart, November 26, 2018 Sven Buechel From Sentiment to Emotion

38

Crosslingual Application

Word Val Aro Dom Joy Ang Sadn Fear Disg sunshine 7.3 3.2 6.8 4.1 1.3 1.2 1.1 1.3 terrorism 1.8 8.1 4.2 1.4 4.1 3.2 3.8 3.6 earthquake 1.9 8.3 1.9 1.2 3.2 3.8 4.3 2.7

(Buechel & Hahn, EACL 2017, CogSci 2017, LREC 2018)

ML features prediction ML features prediction

slide-39
SLIDE 39

Invited Talk at IMS, Universität Stuttgart Stuttgart, November 26, 2018 Sven Buechel From Sentiment to Emotion

39

Crosslingual Application

Word Val Aro Dom Joy Ang Sadn Fear Disg 7.3 3.2 6.8 4.1 1.3 1.2 1.1 1.3 1.8 8.1 4.2 1.4 4.1 3.2 3.8 3.6 1.9 8.3 1.9 1.2 3.2 3.8 4.3 2.7

(Buechel & Hahn, EACL 2017, CogSci 2017, LREC 2018)

ML features prediction ML features prediction

slide-40
SLIDE 40

Invited Talk at IMS, Universität Stuttgart Stuttgart, November 26, 2018 Sven Buechel From Sentiment to Emotion

40

Crosslingual Application

Word Val Aro Dom Joy Ang Sadn Fear Disg

Sonnenschein

7.4 3.1 6.9 4.0 1.2 1.1 1.2 1.4

Terrorismus

1.8 8.2 4.1 1.5 4.0 3.1 3.9 3.7

Erdbeben

1.8 8.1 1.8 1.3 3.3 3.9 4.4 2.8 ML features prediction ML features prediction

(Buechel & Hahn, EACL 2017, CogSci 2017, LREC 2018)

slide-41
SLIDE 41

Invited Talk at IMS, Universität Stuttgart Stuttgart, November 26, 2018 Sven Buechel From Sentiment to Emotion

41

Comparison against Human Reliability

  • Collected 8 double-annotated pairs of datasets (en, es, de, pl)
  • New technique to allow for standardized comparison against

split-half reliability

  • Does the model agree more with gold data than two random

groups of ten people would agree with each other? Ø In over 50% of the cases (including cross-lingual setup): Yes!

(Buechel & Hahn, COLING 2018)

slide-42
SLIDE 42

Invited Talk at IMS, Universität Stuttgart Stuttgart, November 26, 2018 Sven Buechel From Sentiment to Emotion

42

Comparison against Human Reliability

  • Collected 8 double-annotated pairs of datasets (en, es, de, pl)
  • New technique to allow for standardized comparison against

split-half reliability

  • Does the model agree more with gold data than two random

groups of ten people would agree with each other? Ø In over 50% of the cases (including cross-lingual setup): Yes!

r1 r2 r3 r4 r5 r6 i1 i2 i3 i4 i5 i6

(Buechel & Hahn, COLING 2018)

slide-43
SLIDE 43

Invited Talk at IMS, Universität Stuttgart Stuttgart, November 26, 2018 Sven Buechel From Sentiment to Emotion

43

Comparison against Human Reliability

  • Collected 8 double-annotated pairs of datasets (en, es, de, pl)
  • New technique to allow for standardized comparison against

split-half reliability

  • Does the model agree more with gold data than two random

groups of ten people would agree with each other? Ø In over 50% of the cases (including cross-lingual setup): Yes!

r1 r4 r5 i1 i2 i3 i4 i5 i6 r2 r3 r6 i1 i2 i3 i4 i5 i6

(Buechel & Hahn, COLING 2018)

slide-44
SLIDE 44

Invited Talk at IMS, Universität Stuttgart Stuttgart, November 26, 2018 Sven Buechel From Sentiment to Emotion

44

Comparison against Human Reliability

  • Collected 8 double-annotated pairs of datasets (en, es, de, pl)
  • New technique to allow for standardized comparison against

split-half reliability

  • Does the model agree more with gold data than two random

groups of ten people would agree with each other? Ø In over 50% of the cases (including cross-lingual setup): Yes!

i1 i2 i3 i4 i5 i6 i1 i2 i3 i4 i5 i6

(Buechel & Hahn, COLING 2018)

slide-45
SLIDE 45

Invited Talk at IMS, Universität Stuttgart Stuttgart, November 26, 2018 Sven Buechel From Sentiment to Emotion

45

Comparison against Human Reliability

  • Collected 8 double-annotated pairs of datasets (en, es, de, pl)
  • New technique to allow for standardized comparison against

split-half reliability

  • Does the model agree more with gold data than two random

groups of ten people would agree with each other? Ø In over 50% of the cases (also in crosslingual setup): Yes!

(Buechel & Hahn, COLING 2018)

slide-46
SLIDE 46

Invited Talk at IMS, Universität Stuttgart Stuttgart, November 26, 2018 Sven Buechel From Sentiment to Emotion

46

Generating New Emotion Lexicons

  • Identify VA(D) or BE lexicons which do not have

complementary ratings for that language

  • Apply models for prediction
  • Gold quality

Ø New ratings for 13 languages, up to 13k entries each

(en, es, de, pl, it, nl, pt, zh, id, fr, gr, fn, sv) (Buechel & Hahn, COLING 2018)

slide-47
SLIDE 47

Invited Talk at IMS, Universität Stuttgart Stuttgart, November 26, 2018 Sven Buechel From Sentiment to Emotion

47

Interim Conclusion II

  • Multitude of competing emotion representation formats

endangers interoperability

  • Proposed emotion representation mapping
  • Automatically converted ratings are as reliable as gold data
slide-48
SLIDE 48

Invited Talk at IMS, Universität Stuttgart Stuttgart, November 26, 2018 Sven Buechel From Sentiment to Emotion

48

Outline

  • Introduction
  • Applications of emotion analysis in DH and CSS

Ø Dealing with lack of interoperability

  • Dealing with data sparsity
  • Discussion and conclusion
slide-49
SLIDE 49

Invited Talk at IMS, Universität Stuttgart Stuttgart, November 26, 2018 Sven Buechel From Sentiment to Emotion

49

Outline

  • Introduction
  • Applications of emotion analysis in DH and CSS
  • Dealing with lack of interoperability

Ø Dealing with data sparsity

  • Discussion and conclusion
slide-50
SLIDE 50

Invited Talk at IMS, Universität Stuttgart Stuttgart, November 26, 2018 Sven Buechel From Sentiment to Emotion

50

Two Popular Misconceptions about DL?

  • Enormous data requirements

– cf. WASSA 2017 shared task

  • Insufficient affective information in pre-trained embeddings

(Tang et al., 2014)

good bad

!

slide-51
SLIDE 51

Invited Talk at IMS, Universität Stuttgart Stuttgart, November 26, 2018 Sven Buechel From Sentiment to Emotion

51

Word Emotion Induction

Embeddings ML features prediction Word Val Aro Dom Joy Ang Sadn Fear Disg sunshine ?? ?? ?? 4.1 1.3 1.2 1.1 1.3 terrorism ?? ?? ?? 1.4 4.1 3.2 3.8 3.6 earthquake ?? ?? ?? 1.2 3.2 3.8 4.3 2.7 ML

slide-52
SLIDE 52

Invited Talk at IMS, Universität Stuttgart Stuttgart, November 26, 2018 Sven Buechel From Sentiment to Emotion

52

Emotion Lexicons

  • 11 data sets
  • 1 to 14k entries
  • 9 languages

Source ID Language Format # Entries Bradley and Lang (1999) EN English VAD 1,034 Warriner et al. (2013) EN+ English VAD 13,915 Redondo et al. (2007) ES Spanish VAD 1,034 Stadthagen-Gonzalez et al. (2017) ES+ Spanish VA 14,031 Schmidtke et al. (2014) DE German VAD 1,003 Yu et al. (2016a) ZH Chinese VA 2,802 Imbir (2016) PL Polish VAD 4,905 Montefinese et al. (2014) IT Italian VAD 1,121 Soares et al. (2012) PT Portuguese VAD 1,034 Moors et al. (2013) NL Dutch VAD 4,299 Sianipar et al. (2016) ID Indonesian VAD 1,490

slide-53
SLIDE 53

Invited Talk at IMS, Universität Stuttgart Stuttgart, November 26, 2018 Sven Buechel From Sentiment to Emotion

53

Model Details

1 2 3 . . . 300 1 2 3 . . . 256 1 2 . . . 128 1 2 3

  • utput layer

affine transformation two hidden layers shared across VAD .5 dropout LReLU activation embedding layer .2 dropout

slide-54
SLIDE 54

Invited Talk at IMS, Universität Stuttgart Stuttgart, November 26, 2018 Sven Buechel From Sentiment to Emotion

54

Word Embeddings

  • All languages: FastText vectors trained on Wikipedias

(Graves et al., LREC’18)

  • English

– Google News (SGNS, 100B) – Common Crawl (FastText, 600B)

  • Not updated during training
slide-55
SLIDE 55

Invited Talk at IMS, Universität Stuttgart Stuttgart, November 26, 2018 Sven Buechel From Sentiment to Emotion

55

Experimental Setup

  • Compare our model against 5 reference methods

– Linear regression baseline – Similarity to seed words (Turney & Littman, 2003) – Densifier (Rothe & Schütze, 2016) – Ridge regression (Li et al., 2017) – Boosted MLP (Du & Zhang, 2016)

  • Evaluate on 11 data sets
  • 3 distinct embedding models for English
slide-56
SLIDE 56

Invited Talk at IMS, Universität Stuttgart Stuttgart, November 26, 2018 Sven Buechel From Sentiment to Emotion

56

New State-of-the-Art Results

Mean over all conditions

Linear Regression Turney & Littman (2003) Rothe & Schütze (2016) Li et al. (2017) Du & Zhang (2016) Our Work

0.5 0.575 0.65 0.725 0.8

0.73 0.68 0.66 0.61 0.61 0.64

  • Very close to human performance (SHR and ISR)
  • Word embeddings do not contain affective information???

***

(Buechel & Hahn, NAACL 2018)

slide-57
SLIDE 57

Invited Talk at IMS, Universität Stuttgart Stuttgart, November 26, 2018 Sven Buechel From Sentiment to Emotion

57

Sentence-Level EA in Small Datasets

  • How much gold data is needed for sentence-level prediction?
  • Chose four datasets

– between 192 and 1000 instances – English, Polish, Portuguese – VAD and BE

  • Same embeddings models as last study

(Buechel et al., arXiv 2018)

slide-58
SLIDE 58

Invited Talk at IMS, Universität Stuttgart Stuttgart, November 26, 2018 Sven Buechel From Sentiment to Emotion

58

Small Sized Models of Different Architectures

  • Baseline

– BoW Ridge Regression – Bag-of-Vectors Ridge Regression

  • DL models:

Model Filters Recurrent 1st Dense 2nd Dense FFN

  • 256

128 CNN 128

  • 128
  • GRU
  • 128

128

  • LSTM
  • 128

128

  • CNN-LSTM

128 128 128

slide-59
SLIDE 59

Invited Talk at IMS, Universität Stuttgart Stuttgart, November 26, 2018 Sven Buechel From Sentiment to Emotion

59

Results

  • All DL systems did surprisingly well on all datasets
  • GRU performed best by 1%-pt over all datasets
  • Beats (weak) IAA and previous SOTA on SemEval 2007 data

10 20 30 40 50 60 70

Original Winning System (Chaumartin, 2007) IAA SOTA (Beck, IJCNLP 2017) Our GRU

Performance in Pearon's r on SemEval 2007 data

(Buechel et al., EMNLP 2018, arXiv 2018)

slide-60
SLIDE 60

Invited Talk at IMS, Universität Stuttgart Stuttgart, November 26, 2018 Sven Buechel From Sentiment to Emotion

60

Influence of Training Size on Performance

  • GRU feasible down to 300 samples
  • CNN and FFN feasible down to 100 samples

(Buechel et al., arXiv 2018)

slide-61
SLIDE 61

Invited Talk at IMS, Universität Stuttgart Stuttgart, November 26, 2018 Sven Buechel From Sentiment to Emotion

61

Outline

  • Introduction
  • Applications of emotion analysis in DH and CSS
  • Dealing with lack of interoperability

Ø Dealing with data sparsity

  • Discussion and conclusion
slide-62
SLIDE 62

Invited Talk at IMS, Universität Stuttgart Stuttgart, November 26, 2018 Sven Buechel From Sentiment to Emotion

62

Outline

  • Introduction
  • Applications of emotion analysis in DH and CSS
  • Dealing with lack of interoperability
  • Dealing with data sparsity

Ø Discussion and conclusion

slide-63
SLIDE 63

Invited Talk at IMS, Universität Stuttgart Stuttgart, November 26, 2018 Sven Buechel From Sentiment to Emotion

63

Applications of Emotion Analysis

  • Emotion more expressive than sentiment
  • Advantageous in interdisciplinary applications
  • VA(D) seems quite feasible

– general purpose – easy to visualize – good value for money

slide-64
SLIDE 64

Invited Talk at IMS, Universität Stuttgart Stuttgart, November 26, 2018 Sven Buechel From Sentiment to Emotion

64

Dealing with Lack of Interoperability

  • Many different emotion representation formats
  • Endanger interoperability of tools, datasets, and analyses
  • Emotion representation mapping tackles this problem by

allowing to convert between formats

  • Mapped gold data is as reliable as actual gold data, probably

even in cross-lingual applications

slide-65
SLIDE 65

Invited Talk at IMS, Universität Stuttgart Stuttgart, November 26, 2018 Sven Buechel From Sentiment to Emotion

65

Dealing with Data Sparsity

  • Turns out to be surprisingly unproblematic
  • Multi-task learning helps a bit
  • Small models and strong, pre-trained embeddings
  • Word embeddings contain plenty of affective information

(as opposed to popular claims in the literature)

slide-66
SLIDE 66

Invited Talk at IMS, Universität Stuttgart Stuttgart, November 26, 2018 Sven Buechel From Sentiment to Emotion

66

From Sentiment to Emotion:

Challenges of a More Fine-Grained Analysis of Affective Language

Sven Buechel

Slides: https://julielab.de/downloads/publications/slides/buechel_invited_ims_2018.pdf

Jena University Language and Information Engineering (JULIE) Lab Friedrich-Schiller-Universität Jena, Jena, Germany https://julielab.de

slide-67
SLIDE 67

Invited Talk at IMS, Universität Stuttgart Stuttgart, November 26, 2018 Sven Buechel From Sentiment to Emotion

67

References

Sven Buechel, João Sedoc, H. Andrew Schwartz, and Lyle Ungar. 2018. Learning Neural Emotion Analysis from 100 Observations: The Surprising Effectiveness of Pre-Trained Word Representations. In arXiv:1810.10949. Sven Buechel, Anneke Buffone, Barry Slaff, Lyle Ungar, João Sedoc. 2018. Modeling Empathy and Distress in Reaction to News Stories. In EMNLP 2018. Johannes Hellrich, Sven Buechel and Udo Hahn. 2018. JeSemE: A Website for Exploring Diachronic Changes in Word Meaning and Emotion. In COLING 2018: System Demonstrations. Sven Buechel and Udo Hahn. 2018. Emotion Representation Mapping for Automatic Lexicon Construction (Mostly) Performs on Human Level. In COLING 2018. Sebastian G.M. Händschke, Sven Buechel, Jan Goldenstein, Philipp Poschmann, Tinghui Duan, Peter Walgenbach and Udo

  • Hahn. 2018. A Corpus of Corporate Annual and Social Responsibility Reports: 280 Million Tokens of balanced

Organizational Writing. In ECONLP @ ACL 2018. Sven Buechel and Udo Hahn. 2018. Word Emotion Induction for Multiple Languages as a Deep Multi-Task Learning

  • Problem. In NAACL 2018 .

Sven Buechel and Udo Hahn. 2018. Representation Mapping: A Novel Approach to Generate High-Quality Multi-Lingual Emotion Lexicons. In LREC 2018. Sven Buechel, Johannes Hellrich and Udo Hahn: The Course of Emotion in Three Centuries of German Text: A Methodological Framework. In DH 2017. Sven Buechel and Udo Hahn. 2017. A Flexible Mapping Scheme for Discrete and Dimensional Emotion Representations: Evidence from Textual Stimuli. In CogSci 2017. Sven Buechel and Udo Hahn. 2017. EmoBank: Studying the Impact of Annotation Perspective and Representation Format

  • n Dimensional Emotion Analysis. In EACL 2017.

Sven Buechel and Udo Hahn. 2016. Emotion analysis as a regression problem - Dimensional models and their implications

  • n emotion representation and metrical evaluation. In ECAI 2016.

Sven Buechel, Udo Hahn, Jan Goldenstein, Sebastian G. M. Händschke, and Peter Walgenbach. 2016. Do enterprises have emotions? In WASSA @ NAACL 2016.

slide-68
SLIDE 68

Invited Talk at IMS, Universität Stuttgart Stuttgart, November 26, 2018 Sven Buechel From Sentiment to Emotion

68

Backup Slides

slide-69
SLIDE 69

Invited Talk at IMS, Universität Stuttgart Stuttgart, November 26, 2018 Sven Buechel From Sentiment to Emotion

69

Introduction: Sentiment and Emotion

slide-70
SLIDE 70

Invited Talk at IMS, Universität Stuttgart Stuttgart, November 26, 2018 Sven Buechel From Sentiment to Emotion

70

NLP before Sentiment Analysis

  • High-level NLP tasks used to be centered around facts
  • information/relation extraction
  • document classication
  • semantic parsing
  • natural language inference
  • Then, around 2000, something happend...
slide-71
SLIDE 71

Invited Talk at IMS, Universität Stuttgart Stuttgart, November 26, 2018 Sven Buechel From Sentiment to Emotion

71

Growing Interest in Subjective Language

semantic polarity of words

(Hatzivassiloglou & McKeown, 1997)

evaluative statements

(Pang et al., 2002)

expression of feelings

good fantastic great poor mediocre boring The pizza was great! The service was aweful... I just love the peace and quietness after a summer rain. I hate John Doe, he has a terrible sense of humor.

slide-72
SLIDE 72

Invited Talk at IMS, Universität Stuttgart Stuttgart, November 26, 2018 Sven Buechel From Sentiment to Emotion

72

Different „flavors“ of sentiment analysis

  • Polarity prediction (SA as „document classification“)
  • Aspect-based
  • Opinion holder and target identification
  • Related task: detecting subjectivity, irony, empathy, hate

speech, offensive language

slide-73
SLIDE 73

Invited Talk at IMS, Universität Stuttgart Stuttgart, November 26, 2018 Sven Buechel From Sentiment to Emotion

73

Application Domains

  • Product reviews / analytics

– Restaurant (Yelp) – Online retailers (Amazon) – Movies (RottenTomatoes, IMDB)

  • Social media (esp. Twitter)

– Political science – Public relations – Stock market prediction

rottentomatoes.com twitter.com twitter.com

slide-74
SLIDE 74

Invited Talk at IMS, Universität Stuttgart Stuttgart, November 26, 2018 Sven Buechel From Sentiment to Emotion

74

Positive Activation – Negative Activation (PANA)

(Watson & Tellegen, 1985) high arousal low arousal high valence low valence

high positive activation low positive activation high negative activation low negative activation

slide-75
SLIDE 75

Invited Talk at IMS, Universität Stuttgart Stuttgart, November 26, 2018 Sven Buechel From Sentiment to Emotion

75

Lövheim Cube of Emotion

source: https://en.wikipedia.org/wiki/L%C3%B6vheim_cube_of_emotion

(Lövheim, 2012)

slide-76
SLIDE 76

Invited Talk at IMS, Universität Stuttgart Stuttgart, November 26, 2018 Sven Buechel From Sentiment to Emotion

76

Annotation Cost vs. Expressiveness

Expressiveness Annotation Cost

binary polarity ternary polarity numerical Plutchik class-based Ekman numerical Ekman class-based Plutchik VA VAD

slide-77
SLIDE 77

Invited Talk at IMS, Universität Stuttgart Stuttgart, November 26, 2018 Sven Buechel From Sentiment to Emotion

77

Arguments in Favor of Dimensional Models

  • Good value for money
  • General purpose (one set of variables fits all use cases)
  • Large overlap with psychology
  • Interpretability

– Intuitive to understand (in contrast to PANA, Lövheim) – Nice visualizations

slide-78
SLIDE 78

Invited Talk at IMS, Universität Stuttgart Stuttgart, November 26, 2018 Sven Buechel From Sentiment to Emotion

78

Organizational Emotion (WASSA, ECONLP)

slide-79
SLIDE 79

Invited Talk at IMS, Universität Stuttgart Stuttgart, November 26, 2018 Sven Buechel From Sentiment to Emotion

79

JOCO Corpus Statistics

  • 280M Tokens (for comparison: BNC has 100M),
  • 5K reports
  • Equal distribution by country
  • 250K tokens of annual vs. 35K tokens of CSR reports
slide-80
SLIDE 80

Invited Talk at IMS, Universität Stuttgart Stuttgart, November 26, 2018 Sven Buechel From Sentiment to Emotion

80

Results — Organizational Writing vs. News Topics

  • Based on Reuters Corpus Volume 1 (RCV1)
  • 800k newswire documents
  • Hierarchy of 103 topic codes

0.3 0.5 0.7 −1.05 −1.00 −0.95 −0.90 −0.85 Valence Arousal

GCRIM GDEF GDIP GDIS GENV GHEA GPRO GREL GSCI GTOUR GWEA ANN CSR ECAT MCAT CCAT GFAS GSPO

0.2 0.4 0.6 0.8 0.3 0.4 0.5 0.6 0.7 Valence Dominance

GCRIM GDEF GDIP GDIS GENV GHEA GPRO GREL GSCI GTOUR GWEA ECAT MCAT CCAT ANN CSR GFAS GSPO

slide-81
SLIDE 81

Invited Talk at IMS, Universität Stuttgart Stuttgart, November 26, 2018 Sven Buechel From Sentiment to Emotion

81

Historical Emotions (LT4DH, DH, COLING)

slide-82
SLIDE 82

Invited Talk at IMS, Universität Stuttgart Stuttgart, November 26, 2018 Sven Buechel From Sentiment to Emotion

82

Methodological Framework

lexicon

lexicon

adapt + expand apply for emotion analysis modern historically adapted historical text

slide-83
SLIDE 83

Invited Talk at IMS, Universität Stuttgart Stuttgart, November 26, 2018 Sven Buechel From Sentiment to Emotion

83

Target Corpus: DTA

Im DTA verfügbare Werke

nach Genre und Dekade Belletristik Gebrauchsliteratur Wissenschaft 1601ff. 1611ff. 1621ff. 1631ff. 1641ff. 1651ff. 1661ff. 1671ff. 1681ff. 1691ff. 1701ff. 1711ff. 1721ff. 1731ff. 1741ff. 1751ff. 1761ff. 1771ff. 1781ff. 1791ff. 1801ff. 1811ff. 1821ff. 1831ff. 1841ff. 1851ff. 1861ff. 1871ff. 1881ff. 1891ff. 1901ff.

50 100 150

Werke

Documents 1600s 1700s 1800s 1900s Academia Functional Belles lettres

  • 1st third shows different genre distribution
  • Individual decades comprise too little text

Ø Aggregate 30-years slices Ø Select 1690-1899 (~ 1k documents, 7 slices)

http://www.deutsches-textarchiv.de/doku/textauswahl

slide-84
SLIDE 84

Invited Talk at IMS, Universität Stuttgart Stuttgart, November 26, 2018 Sven Buechel From Sentiment to Emotion

84

Target Corpus: DTA

Im DTA verfügbare Werke

nach Genre und Dekade Belletristik Gebrauchsliteratur Wissenschaft 1601ff. 1611ff. 1621ff. 1631ff. 1641ff. 1651ff. 1661ff. 1671ff. 1681ff. 1691ff. 1701ff. 1711ff. 1721ff. 1731ff. 1741ff. 1751ff. 1761ff. 1771ff. 1781ff. 1791ff. 1801ff. 1811ff. 1821ff. 1831ff. 1841ff. 1851ff. 1861ff. 1871ff. 1881ff. 1891ff. 1901ff.

50 100 150

Werke

Documents 1600s 1700s 1800s 1900s Academia Functional Belles lettres

  • 1st third shows different genre distribution
  • Individual decades comprise too little text

Ø Aggregate 30-years slices Ø Select 1690-1899 (~ 1k documents, 7 slices)

http://www.deutsches-textarchiv.de/doku/textauswahl

slide-85
SLIDE 85

Invited Talk at IMS, Universität Stuttgart Stuttgart, November 26, 2018 Sven Buechel From Sentiment to Emotion

85

Distinction of Academic Subclasses

−8 −4 2 4 −4 −2 2 Valence Arousal

−8 −4 2 4 −4 2 4 Valence Dominance −4 −2 2 −4 2 4 Arousal Dominance

Law Philosophy Mathematics Technology Physics

slide-86
SLIDE 86

Invited Talk at IMS, Universität Stuttgart Stuttgart, November 26, 2018 Sven Buechel From Sentiment to Emotion

86

Development of Literary Forms (1690-1719)

  • Lyric

Narrative Drama

−4 2 −3 −1 1

Valence Arousal

−4 2 −3 −1 1 3

Valence Dominance

−3 −1 1 −3 −1 1 3

Dominance Arousal

3

Dominance

3

slide-87
SLIDE 87

Invited Talk at IMS, Universität Stuttgart Stuttgart, November 26, 2018 Sven Buechel From Sentiment to Emotion

87

Development of Literary Forms (1720-1749)

  • Lyric

Narrative Drama

−4 2 −3 −1 1

Valence Arousal

−4 2 −3 −1 1 3

Valence Dominance

−3 −1 1 −3 −1 1 3

Dominance Arousal

3

Dominance

3

slide-88
SLIDE 88

Invited Talk at IMS, Universität Stuttgart Stuttgart, November 26, 2018 Sven Buechel From Sentiment to Emotion

88

Development of Literary Forms (1750-1779)

  • Lyric

Narrative Drama

Valence Valence Dominance

−4 2 −3 −1 1

Valence Arousal

−4 2 −3 −1 1 3

Valence Dominance

−3 −1 1 −3 −1 1 3

Dominance Arousal

3

Dominance

3

slide-89
SLIDE 89

Invited Talk at IMS, Universität Stuttgart Stuttgart, November 26, 2018 Sven Buechel From Sentiment to Emotion

89

Development of Literary Forms (1780-1809)

  • Lyric

Narrative Drama

−4 2 −3 −1 1

Valence Arousal

−4 2 −3 −1 1 3

Valence Dominance

−3 −1 1 −3 −1 1 3

Dominance Arousal

3

Dominance

3

slide-90
SLIDE 90

Invited Talk at IMS, Universität Stuttgart Stuttgart, November 26, 2018 Sven Buechel From Sentiment to Emotion

90

Development of Literary Forms (1810-1839)

  • Lyric

Narrative Drama

−4 2 −3 −1 1

Valence Arousal

−4 2 −3 −1 1 3

Valence Dominance

−3 −1 1 −3 −1 1 3

Dominance Arousal

3

Dominance

3

slide-91
SLIDE 91

Invited Talk at IMS, Universität Stuttgart Stuttgart, November 26, 2018 Sven Buechel From Sentiment to Emotion

91

Development of Literary Forms (1840-1869)

  • Lyric

Narrative Drama

−4 2 −3 −1 1

Valence Arousal

−4 2 −3 −1 1 3

Valence Dominance

−3 −1 1 −3 −1 1 3

Dominance Arousal

3

Dominance

3

slide-92
SLIDE 92

Invited Talk at IMS, Universität Stuttgart Stuttgart, November 26, 2018 Sven Buechel From Sentiment to Emotion

92

Development of Literary Genres (1870-1899)

  • Lyric

Narrative Drama

−4 2 −3 −1 1

Valence Arousal

−4 2 −3 −1 1 3

Valence Dominance

−3 −1 1 −3 −1 1 3

Dominance Arousal

slide-93
SLIDE 93

Invited Talk at IMS, Universität Stuttgart Stuttgart, November 26, 2018 Sven Buechel From Sentiment to Emotion

93

Emotion Representation Mapping (ECAI, EACL, CogSci, LREC, COLING)

slide-94
SLIDE 94

Invited Talk at IMS, Universität Stuttgart Stuttgart, November 26, 2018 Sven Buechel From Sentiment to Emotion

94

Results of JEmAS

  • Outperforms all systems but one

(10 reference systems in total)

– 1st r ≈ .448 Staiano & Guerini (2014) – 2nd r ≈ .419 Our System – 3rd r ≈ .356 Neviarouskaya et al. (2011)

  • State-of-the-art in 3 out of 6 emotional categories

(Buechel & Hahn, ECAI 2016)

slide-95
SLIDE 95

Invited Talk at IMS, Universität Stuttgart Stuttgart, November 26, 2018 Sven Buechel From Sentiment to Emotion

95

Crowdsourcing a Large-Scale VAD Corpus

  • EmoBank (Buechel & Hahn, EACL 2017)
  • 10k sentences with VAD annotation from [1, 5]
  • Comes with two kinds of double-annotation

– Each sentence is annotated according to reader and writer perspective (pilot study was not fully conclusive (Buechel & Hahn, LAW 2017)) – A subset (around 1.2k) has previously been annotated for BE5 (Strapparava & Mihalcea, SemEval 2007)

  • Compare performance of EmoMap against IAA
slide-96
SLIDE 96

Invited Talk at IMS, Universität Stuttgart Stuttgart, November 26, 2018 Sven Buechel From Sentiment to Emotion

96

IAA in the SemEval Dataset

Rater 1 Rater 2 Rater 3 Item 1 Item 2 Item 3 Item 4

  • For each rater

– compute average annotation of remaining raters – compute correlation between this rater and average annotation

  • Average over all raters
  • Weak point of comparison because based on single human
slide-97
SLIDE 97

Invited Talk at IMS, Universität Stuttgart Stuttgart, November 26, 2018 Sven Buechel From Sentiment to Emotion

97

Split-Half Reliability

r1 r2 r3 r4 r5 r6 i1 i2 i3 i4 i5 i6

  • Correlation-based (numerical values)
  • Increasingly popular within CL (Mohammad et al.)
slide-98
SLIDE 98

Invited Talk at IMS, Universität Stuttgart Stuttgart, November 26, 2018 Sven Buechel From Sentiment to Emotion

98

Split-Half Reliability

r1 r4 r5 i1 i2 i3 i4 i5 i6 r2 r3 r6 i1 i2 i3 i4 i5 i6

  • Correlation-based (numerical values)
  • Increasingly popular within CL (Mohammad et al.)
slide-99
SLIDE 99

Invited Talk at IMS, Universität Stuttgart Stuttgart, November 26, 2018 Sven Buechel From Sentiment to Emotion

99

Split-Half Reliability

i1 i2 i3 i4 i5 i6 i1 i2 i3 i4 i5 i6

  • Correlation-based (numerical values)
  • Increasingly popular within CL (Mohammad et al.)
slide-100
SLIDE 100

Invited Talk at IMS, Universität Stuttgart Stuttgart, November 26, 2018 Sven Buechel From Sentiment to Emotion

100

Spearman-Brown Adjustment

  • SHR heavily influenced by number of raters thus not

comparable between studies

  • Solution: Spearman-Brown Adjustment, estimates reliability

r* if number of raters was increased by factor k

r∗ := k r 1 + (k − 1) r

slide-101
SLIDE 101

Invited Talk at IMS, Universität Stuttgart Stuttgart, November 26, 2018 Sven Buechel From Sentiment to Emotion

101

Comparison against Human Reliability

  • Compare model performance against adjusted SHR for 20

raters (arbitrarily chosen but tough comparison)

  • Outperforming adjusted SHR:

Model agrees more with gold data than two random groups

  • f ten people would agree with each other.
slide-102
SLIDE 102

Invited Talk at IMS, Universität Stuttgart Stuttgart, November 26, 2018 Sven Buechel From Sentiment to Emotion

102

Comparison against Human Performance

  • „Monolingual“ Evaluation: 10-CV on one pair of datasets
  • „Crosslingual“ Evaluation: fixed test set, train on all other

languages

Abbrev. VA(D) BE5 Dom? Overlap en 1 Bradley and Lang (1999) Stevenson et al. (2007) 3 1,028 en 2 Warriner et al. (2013) Stevenson et al. (2007) 3 1,027 es 1 Redondo et al. (2007) Ferr´ e et al. (2017) 3 1,012 es 2 Hinojosa et al. (2016b) Hinojosa et al. (2016a) 3 875 es 3 Stadthagen-Gonzalez et al. (2017b) Stadthagen-Gonz´ alez et al. (2017a) 7 10,491 de 1 V˜

  • et al. (2009)

Briesemeister et al. (2011) 7 1,958 pl 1 Riegel et al. (2015) Wierzba et al. (2015) 7 2,902 pl 2 Imbir (2016) Wierzba et al. (2015) 3 1,272

slide-103
SLIDE 103

Invited Talk at IMS, Universität Stuttgart Stuttgart, November 26, 2018 Sven Buechel From Sentiment to Emotion

103

Results: Monolingual

  • Outperforming human reliability in 66% of cases

Val Aro Dom Joy Ang Sad Fea Dsg en 1 .969 .741 .848 .962 .876 .871 .873 .805 en 2 .964 .704 .861 .942 .868 .821 .860 .799 es 1 .974 .771 .863 .957 .854 .833 .869 .752 es 2 .986 .828 .720 .977 .913 .867 .878 .807 es 3 .915 .692 — .846 .839 .857 .842 .744 de 1 .929 .745 — .894 .778 .644 .785 .461 pl 1 .963 .787 — .946 .872 .826 .805 .826 pl 2 .947 .768 .760 .935 .844 .805 .790 .819 Avg. .956 .754 .810 .932 .855 .816 .838 .752

slide-104
SLIDE 104

Invited Talk at IMS, Universität Stuttgart Stuttgart, November 26, 2018 Sven Buechel From Sentiment to Emotion

104

Results: Cross-Lingual

  • Outperforming human reliability in 54% of cases

Val Aro Joy Ang Sad Fea Dsg en 1 .966 .683 .955 .858 .838 .817 .781 en 2 .956 .642 .934 .855 .810 .791 .800 es 1 .973 .692 .951 .786 .802 .782 .682 es 2 .985 .735 .974 .881 .860 .835 .787 es 3 .908 .548 .839 .821 .850 .807 .728 de 1 .927 .708 .889 .767 .618 .760 .458 pl 1 .957 .666 .937 .848 .784 .745 .801 pl 2 .938 .720 .932 .816 .785 .751 .809 Avg. .951 .674 .926 .829 .793 .786 .731

slide-105
SLIDE 105

Invited Talk at IMS, Universität Stuttgart Stuttgart, November 26, 2018 Sven Buechel From Sentiment to Emotion

105

How Important is Dominance anyway?

Not very!

slide-106
SLIDE 106

Invited Talk at IMS, Universität Stuttgart Stuttgart, November 26, 2018 Sven Buechel From Sentiment to Emotion

106

Newly Generated Emotion Ratings

Mth Lng Format Source #Words m en BE5 Warriner et al. (2013) 12,884 m es VAD Stadthagen-Gonz´ alez et al. (2017a) 10,489 m de BE5 V˜

  • et al. (2009)

944 m pl BE5 Imbir (2016) 3,633 c it BE5 Montefinese et al. (2014) 1,121 c pt BE5 Soares et al. (2012) 1,034 c nl BE5 Moors et al. (2013) 4,299 c id BE5 Sianipar et al. (2016) 1,487 c zh BE5 Yu et al. (2016a); Yao et al. (2017) 3,797 c fr BE5 Monnier and Syssau (2014) 1,031 c gr BE5 Palogiannidi et al. (2016) 1,034 c fn BE5 Eilola and Havelka (2010) 210 c sv BE5 Davidson and Innes-Ker (2014) 99