Modeling Empathy and Distress in Reaction to News Stories Sven - - PowerPoint PPT Presentation

▶
modeling empathy and distress in reaction to news stories
SMART_READER_LITE
LIVE PREVIEW

Modeling Empathy and Distress in Reaction to News Stories Sven - - PowerPoint PPT Presentation

Mar 17, 2023 30 likes •426 views

EMNLP 2018 Brussels, Belgium, November 4, 2018 Modeling Empathy and Distress in Reaction to News Stories Sven Buechel 2* Anneke Buffone 1* Barry Slaff 1 Lyle Ungar 1 Joo Sedoc 1 * equal contribution 2 1 JULIE Lab World Well-Being Project

slide-1
SLIDE 1

EMNLP 2018 Brussels, Belgium, November 4, 2018 Sven Buechel, Anneke Buffone, Barry Slaff, Lyle Ungar, and João Sedoc Modeling Empathy and Distress in Reaction to News Stories

1

Sven Buechel 2*

Modeling Empathy and Distress in Reaction to News Stories

Anneke Buffone 1* Barry Slaff 1 Lyle Ungar 1 João Sedoc 1

World Well-Being Project University of Pennsylvania http://www.wwbp.org 1 JULIE Lab Friedrich-Schiller-Universität Jena http://www.julielab.de 2 * equal contribution

slide-2
SLIDE 2

EMNLP 2018 Brussels, Belgium, November 4, 2018 Sven Buechel, Anneke Buffone, Barry Slaff, Lyle Ungar, and João Sedoc Modeling Empathy and Distress in Reaction to News Stories

2 https://xkcd.com/660/ CC-BY-NC 2.5

slide-3
SLIDE 3

EMNLP 2018 Brussels, Belgium, November 4, 2018 Sven Buechel, Anneke Buffone, Barry Slaff, Lyle Ungar, and João Sedoc Modeling Empathy and Distress in Reaction to News Stories

3 https://xkcd.com/660/ CC-BY-NC 2.5

EMPATHY TIPS FOR ENGINEERS

slide-4
SLIDE 4

EMNLP 2018 Brussels, Belgium, November 4, 2018 Sven Buechel, Anneke Buffone, Barry Slaff, Lyle Ungar, and João Sedoc Modeling Empathy and Distress in Reaction to News Stories

4

Why Empathy?

  • Crucial for how we experience the world

and communicate within it

  • Great potential for social media analysis
  • Scarce work on empathy modeling in written language
slide-5
SLIDE 5

EMNLP 2018 Brussels, Belgium, November 4, 2018 Sven Buechel, Anneke Buffone, Barry Slaff, Lyle Ungar, and João Sedoc Modeling Empathy and Distress in Reaction to News Stories

5

Why Empathy?

  • Crucial for how we experience the world

and communicate within it

  • Great potential for social media analysis
  • Scarce work on empathy modeling in written language
slide-6
SLIDE 6

EMNLP 2018 Brussels, Belgium, November 4, 2018 Sven Buechel, Anneke Buffone, Barry Slaff, Lyle Ungar, and João Sedoc Modeling Empathy and Distress in Reaction to News Stories

6

Shortcomings of Prior Work

  • No publicly available gold standard

ØFirst publicly available gold standard (CC-BY)

  • 3rd person ground truth

ØAnnotations by the experiencer

(new annotation methodology)

  • Disconnected from psychological theory and research

Ø Distinguish two different types of empathy

(in line with psych. research)

slide-7
SLIDE 7

EMNLP 2018 Brussels, Belgium, November 4, 2018 Sven Buechel, Anneke Buffone, Barry Slaff, Lyle Ungar, and João Sedoc Modeling Empathy and Distress in Reaction to News Stories

7

Contribution

  • No publicly available gold standard

ØFirst publicly available gold standard (CC-BY)

  • 3rd person ground truth

ØAnnotations by the experiencer

(new annotation methodology)

  • Disconnected from psychological theory and research

Ø Distinguish two different types of empathy

(in line with psych. research)

slide-8
SLIDE 8

EMNLP 2018 Brussels, Belgium, November 4, 2018 Sven Buechel, Anneke Buffone, Barry Slaff, Lyle Ungar, and João Sedoc Modeling Empathy and Distress in Reaction to News Stories

8

Contribution

  • No publicly available gold standard

ØFirst publicly available gold standard (CC-BY)

  • 3rd person ground truth

ØAnnotations by the experiencer

(new annotation methodology)

  • Disconnected from psychological theory and research

Ø Distinguish two different types of empathy

(in line with psych. research)

slide-9
SLIDE 9

EMNLP 2018 Brussels, Belgium, November 4, 2018 Sven Buechel, Anneke Buffone, Barry Slaff, Lyle Ungar, and João Sedoc Modeling Empathy and Distress in Reaction to News Stories

9

Contribution

  • No publicly available gold standard

ØFirst publicly available gold standard (CC-BY)

  • 3rd person ground truth

ØAnnotations by the experiencer

(new annotation methodology)

  • Disconnected from psychological theory and research

Ø Distinguish two different types of empathy

(in line with psych. research)

slide-10
SLIDE 10

EMNLP 2018 Brussels, Belgium, November 4, 2018 Sven Buechel, Anneke Buffone, Barry Slaff, Lyle Ungar, and João Sedoc Modeling Empathy and Distress in Reaction to News Stories

10

Contribution

  • No publicly available gold standard

ØFirst publicly available gold standard (CC-BY)

  • 3rd person ground truth

ØAnnotations by the experiencer

(new annotation methodology)

  • Disconnected from psychological theory and research

ØDistinguish two different types of empathy

(in line with psych. research)

slide-11
SLIDE 11

EMNLP 2018 Brussels, Belgium, November 4, 2018 Sven Buechel, Anneke Buffone, Barry Slaff, Lyle Ungar, and João Sedoc Modeling Empathy and Distress in Reaction to News Stories

11

Empathic Concern vs. Personal Distress

  • Consensus in psych. that there are multiple forms of empathy
  • We follow most popular distinction by Batson et al. (1987)

– Empathic Concern

positive, other-focused

– Personal Distress

negative, self-focused

  • Post-hoc analysis shows both are distinct (r=.45)
slide-12
SLIDE 12

EMNLP 2018 Brussels, Belgium, November 4, 2018 Sven Buechel, Anneke Buffone, Barry Slaff, Lyle Ungar, and João Sedoc Modeling Empathy and Distress in Reaction to News Stories

12

Empathic Concern vs. Personal Distress

  • Consensus in psych. that there are multiple forms of empathy
  • We follow most popular distinction by Batson et al. (1987)

– Empathic Concern

positive, other-focused

– Personal Distress

negative, self-focused

  • Post-hoc analysis shows that both are distinct (r=.45)
slide-13
SLIDE 13

EMNLP 2018 Brussels, Belgium, November 4, 2018 Sven Buechel, Anneke Buffone, Barry Slaff, Lyle Ungar, and João Sedoc Modeling Empathy and Distress in Reaction to News Stories

13

Problem Definition

Message Empathy Distress I‘m sorry to hear about Dakota‘s parents. [...] 4.8 3.1

  • Given a natural language utterance ...
  • ... predict empathy and distress of the writer from [1, 7]
slide-14
SLIDE 14

EMNLP 2018 Brussels, Belgium, November 4, 2018 Sven Buechel, Anneke Buffone, Barry Slaff, Lyle Ungar, and João Sedoc Modeling Empathy and Distress in Reaction to News Stories

14

Annotation Methodology

Traditional

slide-15
SLIDE 15

EMNLP 2018 Brussels, Belgium, November 4, 2018 Sven Buechel, Anneke Buffone, Barry Slaff, Lyle Ungar, and João Sedoc Modeling Empathy and Distress in Reaction to News Stories

15

Annotation Methodology

Traditional r1 r2 r3 rfinal

slide-16
SLIDE 16

EMNLP 2018 Brussels, Belgium, November 4, 2018 Sven Buechel, Anneke Buffone, Barry Slaff, Lyle Ungar, and João Sedoc Modeling Empathy and Distress in Reaction to News Stories

16

Annotation Methodology

Traditional r1 r2 r3 rfinal Experiencer

  • Annotators “guess“

experiencer‘s feelings

  • Models biased

towards annotators

slide-17
SLIDE 17

EMNLP 2018 Brussels, Belgium, November 4, 2018 Sven Buechel, Anneke Buffone, Barry Slaff, Lyle Ungar, and João Sedoc Modeling Empathy and Distress in Reaction to News Stories

17

Annotation Methodology

Traditional Proposed r1 r2 r3 Stimulus Experiencer rfinal Experiencer

  • Annotation as psych.

experiment

  • Experiencer produces

message and ratings

  • Reliable through multi-

item scales

  • Annotators “guess“

experiencer‘s feelings

  • Models biased

towards annotators

slide-18
SLIDE 18

EMNLP 2018 Brussels, Belgium, November 4, 2018 Sven Buechel, Anneke Buffone, Barry Slaff, Lyle Ungar, and João Sedoc Modeling Empathy and Distress in Reaction to News Stories

18

Annotation Methodology

Traditional Proposed r1 r2 r3 Stimulus Experiencer rfinal r Experiencer

  • Annotation as psych.

experiment

  • Experiencer produces

message and rating

  • Reliable through multi-

item scales

  • Annotators “guess“

experiencer‘s feelings

  • Models biased

towards annotators

slide-19
SLIDE 19

EMNLP 2018 Brussels, Belgium, November 4, 2018 Sven Buechel, Anneke Buffone, Barry Slaff, Lyle Ungar, and João Sedoc Modeling Empathy and Distress in Reaction to News Stories

19

Annotation Methodology

Traditional Proposed r1 r2 r3 Stimulus Experiencer rfinal r Experiencer

  • Annotation as psych.

experiment

  • Experiencer produces

message and rating

  • Reliable through multi-

item scales

  • Annotators “guess“

experiencer‘s feelings

  • Models biased

towards annotators

slide-20
SLIDE 20

EMNLP 2018 Brussels, Belgium, November 4, 2018 Sven Buechel, Anneke Buffone, Barry Slaff, Lyle Ungar, and João Sedoc Modeling Empathy and Distress in Reaction to News Stories

20

Annotation Methodology

Traditional Proposed r1 r2 r3 Stimulus Experiencer rfinal r1 r2 r3 rfinal Experiencer

  • Annotation as psych.

experiment

  • Experiencer produces

message and rating

  • Reliable through

multi-item scales

  • Annotators “guess“

experiencer‘s feelings

  • Models biased

towards annotators

slide-21
SLIDE 21

EMNLP 2018 Brussels, Belgium, November 4, 2018 Sven Buechel, Anneke Buffone, Barry Slaff, Lyle Ungar, and João Sedoc Modeling Empathy and Distress in Reaction to News Stories

21

Use of Multi-Item Scales

slide-22
SLIDE 22

EMNLP 2018 Brussels, Belgium, November 4, 2018 Sven Buechel, Anneke Buffone, Barry Slaff, Lyle Ungar, and João Sedoc Modeling Empathy and Distress in Reaction to News Stories

22

Use of Multi-Item Scales

Mean Empathic Concern

slide-23
SLIDE 23

EMNLP 2018 Brussels, Belgium, November 4, 2018 Sven Buechel, Anneke Buffone, Barry Slaff, Lyle Ungar, and João Sedoc Modeling Empathy and Distress in Reaction to News Stories

23

Corpus Creation Process

crowd workers Qualtrics

  • nline survey

final corpus stimulus

upload manual review take

1680 (M, E, D)-triples

418 online news articles

  • read articles
  • rate Empathy/Distress
  • write Message

(300 to 800 chars)

slide-24
SLIDE 24

EMNLP 2018 Brussels, Belgium, November 4, 2018 Sven Buechel, Anneke Buffone, Barry Slaff, Lyle Ungar, and João Sedoc Modeling Empathy and Distress in Reaction to News Stories

24

Corpus Creation Process

crowd workers Qualtrics

  • nline survey

final corpus stimulus

upload manual review take

1680 (M, E, D)-triples

418 online news articles

  • read articles
  • rate Empathy/Distress
  • write Message

(300 to 800 chars)

Split-half reliability around r=.9 for both empathy and distress J

slide-25
SLIDE 25

EMNLP 2018 Brussels, Belgium, November 4, 2018 Sven Buechel, Anneke Buffone, Barry Slaff, Lyle Ungar, and João Sedoc Modeling Empathy and Distress in Reaction to News Stories

25

Baseline Models for Empathy and Distress

  • Models

– Ridge Regression – Feed-Forward Net – Convolutional Neural Net (1 conv layer, filter sizes 1-3, 100 channels each) – RNN-type architectures did not work because of long sequences

  • Features: FastText embeddings pre-trained on Common Crawl
  • 10-fold cross-validation
  • Correlation values around r = .4
  • Ridge is viable but CNN significantly outperforms the others
slide-26
SLIDE 26

EMNLP 2018 Brussels, Belgium, November 4, 2018 Sven Buechel, Anneke Buffone, Barry Slaff, Lyle Ungar, and João Sedoc Modeling Empathy and Distress in Reaction to News Stories

26

Baseline Models for Empathy and Distress

  • Models

– Ridge Regression – Feed-Forward Net – Convolutional Neural Net (1 conv layer, filter sizes 1-3, 100 channels each) – RNN-type architectures did not work because of long sequences

  • Features: FastText embeddings pre-trained on Common Crawl
  • 10-fold cross-validation
  • Correlation values around r = .4
  • Ridge is viable but CNN significantly outperforms the others
slide-27
SLIDE 27

EMNLP 2018 Brussels, Belgium, November 4, 2018 Sven Buechel, Anneke Buffone, Barry Slaff, Lyle Ungar, and João Sedoc Modeling Empathy and Distress in Reaction to News Stories

27

Baseline Models for Empathy and Distress

  • Models

– Ridge Regression – Feed-Forward Net – Convolutional Neural Net (1 conv layer, filter sizes 1-3, 100 channels each) – RNN-type architectures did not work because of long sequences

  • Features: FastText embeddings pre-trained on Common Crawl
  • 10-fold cross-validation
  • Correlation values around r = .4
  • Ridge is viable but CNN significantly outperforms the others
slide-28
SLIDE 28

EMNLP 2018 Brussels, Belgium, November 4, 2018 Sven Buechel, Anneke Buffone, Barry Slaff, Lyle Ungar, and João Sedoc Modeling Empathy and Distress in Reaction to News Stories

28

Conclusion

  • Social media offers great opportunity to study empathy
  • Modeling empathy received little attention for written language
  • We presented the first publicly available gold standard

https://github.com/wwbp/empathic_reactions

  • We distinguish Empathic Concern and Personal Distress
  • New annotation methodology collects reliable ratings

from the experiencer using multi-item scales

slide-29
SLIDE 29

EMNLP 2018 Brussels, Belgium, November 4, 2018 Sven Buechel, Anneke Buffone, Barry Slaff, Lyle Ungar, and João Sedoc Modeling Empathy and Distress in Reaction to News Stories

29

Sven Buechel *

Modeling Empathy and Distress in Reaction to News Stories

Anneke Buffone * Barry Slaff Lyle Ungar João Sedoc

https://xkcd.com/660/ CC-BY-NC 2.5

EMPATHY TIPS FOR ENGINEERS

slide-30
SLIDE 30

EMNLP 2018 Brussels, Belgium, November 4, 2018 Sven Buechel, Anneke Buffone, Barry Slaff, Lyle Ungar, and João Sedoc Modeling Empathy and Distress in Reaction to News Stories

30

Backup Slides

slide-31
SLIDE 31

EMNLP 2018 Brussels, Belgium, November 4, 2018 Sven Buechel, Anneke Buffone, Barry Slaff, Lyle Ungar, and João Sedoc Modeling Empathy and Distress in Reaction to News Stories

31

Previous Work

spoken written What we talk about today Xiao et al. (2012) Gibson et al. (2015) Khanpour et al. (2017)

slide-32
SLIDE 32

EMNLP 2018 Brussels, Belgium, November 4, 2018 Sven Buechel, Anneke Buffone, Barry Slaff, Lyle Ungar, and João Sedoc Modeling Empathy and Distress in Reaction to News Stories

32

Exemplary Entries

Empathy Distress Message 4.8 3.1 I‘m sorry to hear about Dakota‘s parents. No one wants that to happen and it‘s unfortunate that her parents couldn‘t work it out. I hope they are able to still remain civil around the kids and family. [...] 4.0 5.5 Here‘s an article about [a] crazed person who murdered two unfortunate women overseas. Life is crazy. [...] It feels like there‘s on place safe in this world to be a woman sometimes. 1.0 1.3 I just read an article about some chowder-head who used a hammer and a pick ax to destroy Donald Trump‘s star

  • n the Hollywood walk of fame. [...] Lol, can you believe

this garbage? Who has such a hollow and pathetic life that they don‘t have anything better to do with their time than commit petty vandalism because they dislike some politician? [...]

slide-33
SLIDE 33

EMNLP 2018 Brussels, Belgium, November 4, 2018 Sven Buechel, Anneke Buffone, Barry Slaff, Lyle Ungar, and João Sedoc Modeling Empathy and Distress in Reaction to News Stories

33

Split-Half Reliability (SHR)

  • Based on Pearson correlation r
  • Flexible (works with crowdsourcing and best-worst scaling)
  • Most popular in psychology
  • Increasingly popular within CL (Mohammad et al.)

i1 i2 i3 i4 i5 i6 d1 d2 d3 d4 d5 d6

slide-34
SLIDE 34

EMNLP 2018 Brussels, Belgium, November 4, 2018 Sven Buechel, Anneke Buffone, Barry Slaff, Lyle Ungar, and João Sedoc Modeling Empathy and Distress in Reaction to News Stories

34

Split-Half Reliability (SHR)

i1 i4 i5 d1 d2 d3 d4 d5 d6 i2 i3 i6 d1 d2 d3 d4 d5 d6

  • Based on Pearson correlation r
  • Flexible (works with crowdsourcing and best-worst scaling)
  • Most popular in psychology
  • Increasingly popular within CL (Mohammad et al.)
slide-35
SLIDE 35

EMNLP 2018 Brussels, Belgium, November 4, 2018 Sven Buechel, Anneke Buffone, Barry Slaff, Lyle Ungar, and João Sedoc Modeling Empathy and Distress in Reaction to News Stories

35

Split-Half Reliability (SHR)

d1 d2 d3 d4 d5 d6 d1 d2 d3 d4 d5 d6

  • Based on Pearson correlation r
  • Flexible (works with crowdsourcing and best-worst scaling)
  • Most popular in psychology
  • Increasingly popular within CL (Mohammad et al.)
slide-36
SLIDE 36

EMNLP 2018 Brussels, Belgium, November 4, 2018 Sven Buechel, Anneke Buffone, Barry Slaff, Lyle Ungar, and João Sedoc Modeling Empathy and Distress in Reaction to News Stories

36

Bivariate Rating Distribution

  • Good coverage of the full range of rating scales
  • Empathy and distress are distinct (moderate correlation of r=.45)
slide-37
SLIDE 37

EMNLP 2018 Brussels, Belgium, November 4, 2018 Sven Buechel, Anneke Buffone, Barry Slaff, Lyle Ungar, and João Sedoc Modeling Empathy and Distress in Reaction to News Stories

37

Models

  • Ridge Regression
  • Feed-Forward Net

– two layers, 256 and 128 units

  • Convolutional Neural Net

– one conv layer – filter sizes 1, 2, 3 – 100 output channels – average pooling – dense layer (128)

slide-38
SLIDE 38

EMNLP 2018 Brussels, Belgium, November 4, 2018 Sven Buechel, Anneke Buffone, Barry Slaff, Lyle Ungar, and João Sedoc Modeling Empathy and Distress in Reaction to News Stories

38

Experimental Setup

  • Features: FastText embeddings pre-trained on Common Crawl
  • Train distinct models for empathy and distress
  • Exclude 10% of data for dev experiments
  • 10-fold CV on remaining data
  • Evaluate with Pearson correlation r
slide-39
SLIDE 39

EMNLP 2018 Brussels, Belgium, November 4, 2018 Sven Buechel, Anneke Buffone, Barry Slaff, Lyle Ungar, and João Sedoc Modeling Empathy and Distress in Reaction to News Stories

39

Results

  • Ridge regression is viable option, outperforms FFN
  • CNN significantly outperforms Ridge and FFN

(* two-tailed paired t-test; p < .05)

Empathy Distress Mean Ridge .385 .410 .398 FFN .379 .401 .390 CNN .404* .444* .424*