Linguistic Harbingers of Betrayal me, Vlad Niculae , Cornell - - PowerPoint PPT Presentation

▶
linguistic harbingers of betrayal
SMART_READER_LITE
LIVE PREVIEW

Linguistic Harbingers of Betrayal me, Vlad Niculae , Cornell - - PowerPoint PPT Presentation

Dec 12, 2023 250 likes •1.06k views

Linguistic Harbingers of Betrayal me, Vlad Niculae , Cornell University with Srijan Kumar , University of Maryland College Park Jordan Boyd-Graber , University of Colorado Boulder and Cristian Danescu-Niculescu-Mizil

slide-1
SLIDE 1

Linguistic
 Harbingers


  • f Betrayal

Vlad Niculae, Cornell University Srijan Kumar, University of Maryland College Park Jordan Boyd-Graber, University of Colorado Boulder Cristian Danescu-Niculescu-Mizil Cornell University me,
 with
 
 and

slide-2
SLIDE 2

Linguistic
 Harbingers


  • f Betrayal

is everywhere.

slide-3
SLIDE 3
slide-4
SLIDE 4

Are there any 
 linguistic cues that foretell betrayal?

slide-5
SLIDE 5

What is betrayal? (And how does it
 differ from just lying?)

slide-6
SLIDE 6

Can this be 
 a betrayal?

slide-7
SLIDE 7

Can this be 
 a betrayal?

Deceptive review spam
 (Li, Huang, Yang & Zhu, 2011)
 (Ott, Choi, Cardie & Hancock, 2011)
 (Feng, Banerjee & Choi, 2012)
 …
slide-8
SLIDE 8

Can this be 
 a betrayal?

Deceptive review spam
 (Li, Huang, Yang & Zhu, 2011)
 (Ott, Choi, Cardie & Hancock, 2011)
 (Feng, Banerjee & Choi, 2012)
 … Deception in court cases
 (Bachenko, Fitzpatrick 
 & Schonwetter, 2008)
 (Fornaciari & Poesio, 2013)
 …
slide-9
SLIDE 9

Can this be 
 a betrayal?

Deceptive review spam
 (Li, Huang, Yang & Zhu, 2011)
 (Ott, Choi, Cardie & Hancock, 2011)
 (Feng, Banerjee & Choi, 2012)
 … Deception in court cases
 (Bachenko, Fitzpatrick 
 & Schonwetter, 2008)
 (Fornaciari & Poesio, 2013)
 … Elicited deception in essays
 (Newman, Pennebaker, Berry
 & Richards, 2003)
 (Mihalcea & Strapparava, 2009)
 (Pérez-Rosas & Mihalcea, 2014)
 …
slide-10
SLIDE 10

Okay, how about this, can this be a betrayal?

(Feldman and Happ, 2002)
 (Hancock, Curry, Goorha & Woodworth, 2011)
 …
slide-11
SLIDE 11

? t

slide-12
SLIDE 12

“ T h e g a m e t h a t 
 r u i n s f r i e n d s h i p s ”

slide-13
SLIDE 13

“ T h e g a m e t h a t 
 r u i n s f r i e n d s h i p s ”

X

slide-14
SLIDE 14

“ T h e g a m e t h a t 
 r u i n s f r i e n d s h i p s ”

slide-15
SLIDE 15

“ T h e g a m e t h a t 
 r u i n s f r i e n d s h i p s ”

  • n

l i n e !

slide-16
SLIDE 16

“ T h e g a m e t h a t 
 r u i n s f r i e n d s h i p s ”

  • n

l i n e ! 2 4 9 g a m e s ~ 6 m

  • n

t h s / g a m e 
 1 4 5 k m e s s a g e s

d i p l

  • m

.

  • r

g ; u s a k . a s c i i k i n g . c

  • m
slide-17
SLIDE 17
slide-18
SLIDE 18
slide-19
SLIDE 19
slide-20
SLIDE 20
slide-21
SLIDE 21
slide-22
SLIDE 22

help?

slide-23
SLIDE 23


 What I would like you to do is keep Turkey busy and somehow get Russia and Turkey to engage. Meanwhile we need to take VIE, suggest you support me in there.

F

help?

slide-24
SLIDE 24


 What I would like you to do is keep Turkey busy and somehow get Russia and Turkey to engage. Meanwhile we need to take VIE, suggest you support me in there. 
 It's a sensible plan. I'll support you as requested. Please be sure to simultaneously attack SWE.

F

OK

slide-25
SLIDE 25


 What I would like you to do is keep Turkey busy and somehow get Russia and Turkey to engage. Meanwhile we need to take VIE, suggest you support me in there. 
 It's a sensible plan. I'll support you as requested. Please be sure to simultaneously attack SWE.

F F

slide-26
SLIDE 26


 What I would like you to do is keep Turkey busy and somehow get Russia and Turkey to engage. Meanwhile we need to take VIE, suggest you support me in there. 
 It's a sensible plan. I'll support you as requested. Please be sure to simultaneously attack SWE.

F F

slide-27
SLIDE 27


 What I would like you to do is keep Turkey busy and somehow get Russia and Turkey to engage. Meanwhile we need to take VIE, suggest you support me in there. 
 It's a sensible plan. I'll support you as requested. Please be sure to simultaneously attack SWE. …
 stabs !
 
 


F F E

slide-28
SLIDE 28

NOW STAND BACK, I GOTTA PRACTICE MY STABBIN'

slide-29
SLIDE 29

F F E


 What I would like you to do is keep Turkey busy and somehow get Russia and Turkey to engage. Meanwhile we need to take VIE, suggest you support me in there. 
 It's a sensible plan. I'll support you as requested. Please be sure to simultaneously attack SWE. …
 stabs !
 
 


slide-30
SLIDE 30

F F E


 What I would like you to do is keep Turkey busy and somehow get Russia and Turkey to engage. Meanwhile we need to take VIE, suggest you support me in there. 
 It's a sensible plan. I'll support you as requested. Please be sure to simultaneously attack SWE. …
 stabs !
 
 


slide-31
SLIDE 31


 What I would like you to do is keep Turkey busy and somehow get Russia and Turkey to engage. Meanwhile we need to take VIE, suggest you support me in there. 
 It's a sensible plan. I'll support you as requested. Please be sure to simultaneously attack SWE. …
 stabs !
 
 
 Not really sure what to say, except 
 that I regret you did what you did.

F F E

slide-32
SLIDE 32

Curse your sudden
 but inevitable
 betrayal!

slide-33
SLIDE 33


 What I would like you to do is keep Turkey busy and somehow get Russia and Turkey to engage. Meanwhile we need to take VIE, suggest you support me in there 
 It's a sensible plan. I'll support you as requested. Please be sure to simultaneously attack SWE.
 stabs !
 
 
 Not really sure what to say, except 
 that I regret you did what you did.

F F E E

slide-34
SLIDE 34

Identifying Betrayals

(So That We Can Analyze Their Language)

slide-35
SLIDE 35

F F E F E} E E}

Betrayal: friendship followed by
 ≥ two attacks.

Identifying Betrayals

Solid friendships: established (≥ two supports),
 mutual (≥ one from each).

slide-36
SLIDE 36

F F E F E} E E}

250 such betrayals in our Diplomacy dataset. Betrayal: friendship followed by
 ≥ two attacks.

Identifying Betrayals

Solid friendships: established (≥ two supports),
 mutual (≥ one from each).

slide-37
SLIDE 37

F F E F E}

Solid friendships: established (≥ two supports),
 mutual (≥ one from each).

E E F

}

F F F F F

{ }

Matched for: length,
 game year.

Matching Friendship

Betrayal. 250 such betrayals in our Diplomacy dataset.
 We find 250 matching friendships. (9660 msgs., 59 words/msg.)

slide-38
SLIDE 38

F F E F E}

Solid friendships: established (≥ two supports),
 mutual (≥ one from each).

E E F

}

F F F F F

{

Matched
 friendship: no attacks
 (ever).

} }

Matched for: length,
 game year.

Matching Friendship

Betrayal. 250 such betrayals in our Diplomacy dataset.
 We find 250 matching friendships. (9660 msgs., 59 words/msg.)

slide-39
SLIDE 39

F F E F E E E F F F F F F

Matching Friendship

250 such betrayals in our Diplomacy dataset.
 We find 250 matching friendships. (9660 msgs., 59 words/msg.)

slide-40
SLIDE 40

F F E F E E E F F F F F F

Matching Friendship

Linguistic signs of betrayal
 while they act as friends? 250 such betrayals in our Diplomacy dataset.
 We find 250 matching friendships. (9660 msgs., 59 words/msg.)

slide-41
SLIDE 41

F F E F E E E F F F F F F

Matching Friendship

Linguistic signs of betrayal
 while they act as friends? 250 such betrayals in our Diplomacy dataset.
 We find 250 matching friendships. (9660 msgs., 59 words/msg.) The betrayers actively hide it.
 The victims didn’t see it coming.

slide-42
SLIDE 42

Insight:
 Conversational Balance

slide-43
SLIDE 43

Insight:
 Conversational Balance

slide-44
SLIDE 44

Insight:
 Conversational Balance

  • Stable marriages are balanced (Gottman, 1993).
  • So are effective pair programming teams


(Jung, Chong & Leifer, 2012).

slide-45
SLIDE 45

Insight:
 Conversational Balance

  • Stable marriages are balanced (Gottman, 1993).
  • So are effective pair programming teams


(Jung, Chong & Leifer, 2012).

  • Can we apply this to linguistic conversational features?
slide-46
SLIDE 46

(Im)balance: Sentiment

(Proportion of sentences showing positive sentiment.)
 (Error bars show standard error.)

betrayal 0.00 0.01 0.02 0.03

| imbalance |

Imbalance: f(betrayer) - f(victim)

slide-47
SLIDE 47

(Im)balance: Sentiment

(Proportion of sentences showing positive sentiment.)
 (Error bars show standard error.)

matched friendship betrayal 0.00 0.01 0.02 0.03

| imbalance |

Imbalance: f(betrayer) - f(victim)

slide-48
SLIDE 48

(Im)balance: Sentiment

(Proportion of sentences showing positive sentiment.)
 (Error bars show standard error.)

matched friendship betrayal 0.00 0.01 0.02 0.03

| imbalance |

matched friendship betrayer victim 0.00 0.05 0.10 0.15 0.20

amount

Imbalance: f(betrayer) - f(victim)

slide-49
SLIDE 49

(Im)balance: Politeness

(Average 0-1 politeness score of requests: http://politeness.mpi-sws.org)
 (Error bars show standard error.)

matched friendship betrayal 0.00 0.01 0.02 0.03

| imbalance |

matched friendship betrayer victim 0.50 0.55 0.60 0.65 0.70

amount

Imbalance: f(betrayer) - f(victim)

slide-50
SLIDE 50

(Im)balance: Future Planning

(Average number of planning connectors per message, e.g. “next”, “after”)
 (Error bars show standard error.)

matched friendship betrayal 0.00 0.01 0.02 0.03

| imbalance |

matched friendship betrayer victim 0.00 0.02 0.04 0.06 0.08 0.10 0.12

amount

Imbalance: f(betrayer) - f(victim)

slide-51
SLIDE 51

(Im)balance: Future Planning

(Average number of planning connectors per message, e.g. “next”, “after”)
 (Error bars show standard error.)

matched friendship betrayal 0.00 0.01 0.02 0.03

| imbalance |

matched friendship betrayer victim 0.00 0.02 0.04 0.06 0.08 0.10 0.12

amount

Imbalance: f(betrayer) - f(victim) Demand-Withdraw pattern pre-divorce.


(Gottman & Levenson, 2000)
slide-52
SLIDE 52


 (Error bars show standard error.)

Friendships that break exhibit imbalance through language cues.

Conversational (Im)balance

positive sentiment −0.01 0.00 0.01 0.02 0.03 | imbalance | politeness planning matched friendship betrayal

Imbalance: f(betrayer) - f(victim)

slide-53
SLIDE 53

Are backstabbing friendships doomed 
 from the start?

slide-54
SLIDE 54

Are backstabbing friendships doomed 
 from the start?

Or do the dynamics change over time?

slide-55
SLIDE 55

(Im)balance Over Time


 (Error bars show standard error.)

4 and up 3 2 1 betrayal −0.10 −0.05 0.00 0.05 0.10

imbalance time until betrayal Imbalance: f(betrayer) - f(victim). Looking only at betrayals.

slide-56
SLIDE 56


 (Error bars show standard error.) positive sentiment

4 and up 3 2 1 betrayal −0.10 −0.05 0.00 0.05 0.10 4 and up 3 2 1 betrayal −0.10 −0.05 0.00 0.05 0.10 4 and up 3 2 1 betrayal −0.10 −0.05 0.00 0.05 0.10

politeness future planning imbalance time until betrayal Imbalance: f(betrayer) - f(victim)

(Im)balance Over Time

slide-57
SLIDE 57


 (Error bars show standard error.) positive sentiment

4 and up 3 2 1 betrayal −0.10 −0.05 0.00 0.05 0.10

politeness future planning imbalance time until betrayal

4 and up 3 2 1 betrayal −0.10 −0.05 0.00 0.05 0.10 4 and up 3 2 1 betrayal −0.10 −0.05 0.00 0.05 0.10

Imbalance: f(betrayer) - f(victim)

(Im)balance Over Time

slide-58
SLIDE 58


 (Error bars show standard error.) positive sentiment

4 and up 3 2 1 betrayal −0.10 −0.05 0.00 0.05 0.10

politeness future planning imbalance time until betrayal

4 and up 3 2 1 betrayal −0.10 −0.05 0.00 0.05 0.10 4 and up 3 2 1 betrayal −0.10 −0.05 0.00 0.05 0.10

Imbalance: f(betrayer) - f(victim)

(Im)balance Over Time

slide-59
SLIDE 59


 (Error bars show standard error.) positive sentiment

4 and up 3 2 1 betrayal −0.10 −0.05 0.00 0.05 0.10

politeness future planning imbalance time until betrayal

4 and up 3 2 1 betrayal −0.10 −0.05 0.00 0.05 0.10 4 and up 3 2 1 betrayal −0.10 −0.05 0.00 0.05 0.10

Imbalance: f(betrayer) - f(victim)

(Im)balance Over Time

slide-60
SLIDE 60


 (Error bars show standard error.) positive sentiment

4 and up 3 2 1 betrayal −0.10 −0.05 0.00 0.05 0.10

politeness future planning time until betrayal imbalance

4 and up 3 2 1 betrayal 0.13 0.17 0.21 0.25 betrayer victim

amount

4 and up 3 2 1 betrayal −0.10 −0.05 0.00 0.05 0.10 4 and up 3 2 1 betrayal −0.10 −0.05 0.00 0.05 0.10

Imbalance: f(betrayer) - f(victim)

(Im)balance Over Time

slide-61
SLIDE 61


 (Error bars show standard error.) positive sentiment

4 and up 3 2 1 betrayal −0.10 −0.05 0.00 0.05 0.10

politeness future planning time until betrayal imbalance

4 and up 3 2 1 betrayal 0.13 0.17 0.21 0.25 betrayer victim

amount

4 and up 3 2 1 betrayal −0.10 −0.05 0.00 0.05 0.10 4 and up 3 2 1 betrayal −0.10 −0.05 0.00 0.05 0.10

Imbalance: f(betrayer) - f(victim)

(Im)balance Over Time

slide-62
SLIDE 62


 (Error bars show standard error.) positive sentiment

4 and up 3 2 1 betrayal −0.10 −0.05 0.00 0.05 0.10

politeness future planning time until betrayal imbalance

4 and up 3 2 1 betrayal 0.13 0.17 0.21 0.25 betrayer victim

amount

4 and up 3 2 1 betrayal −0.10 −0.05 0.00 0.05 0.10 4 and up 3 2 1 betrayal −0.10 −0.05 0.00 0.05 0.10 4 and up 3 2 1 betrayal 0.60 0.65 0.70

Imbalance: f(betrayer) - f(victim)

(Im)balance Over Time

slide-63
SLIDE 63


 (Error bars show standard error.) positive sentiment

4 and up 3 2 1 betrayal −0.10 −0.05 0.00 0.05 0.10

politeness future planning time until betrayal imbalance

4 and up 3 2 1 betrayal 0.13 0.17 0.21 0.25 betrayer victim

amount

4 and up 3 2 1 betrayal −0.10 −0.05 0.00 0.05 0.10 4 and up 3 2 1 betrayal −0.10 −0.05 0.00 0.05 0.10 4 and up 3 2 1 betrayal 0.60 0.65 0.70

Demand-Withdraw pattern pre-divorce.


(Gottman & Levenson, 2000)

Imbalance: f(betrayer) - f(victim)

(Im)balance Over Time

slide-64
SLIDE 64


 (Error bars show standard error.) positive sentiment

4 and up 3 2 1 betrayal −0.10 −0.05 0.00 0.05 0.10

politeness future planning time until betrayal imbalance

4 and up 3 2 1 betrayal 0.13 0.17 0.21 0.25 betrayer victim

amount

4 and up 3 2 1 betrayal −0.10 −0.05 0.00 0.05 0.10 4 and up 3 2 1 betrayal −0.10 −0.05 0.00 0.05 0.10 4 and up 3 2 1 betrayal 0.60 0.65 0.70

Demand-Withdraw pattern pre-divorce.


(Gottman & Levenson, 2000)

Imbalance: f(betrayer) - f(victim)

(Im)balance Over Time

slide-65
SLIDE 65


 (Error bars show standard error.) positive sentiment

4 and up 3 2 1 betrayal −0.10 −0.05 0.00 0.05 0.10

politeness future planning time until betrayal imbalance

4 and up 3 2 1 betrayal 0.13 0.17 0.21 0.25 betrayer victim

amount

4 and up 3 2 1 betrayal −0.10 −0.05 0.00 0.05 0.10 4 and up 3 2 1 betrayal −0.10 −0.05 0.00 0.05 0.10 4 and up 3 2 1 betrayal 0.60 0.65 0.70 4 and up 3 2 1 betrayal 0.05 0.10 0.15

Demand-Withdraw pattern pre-divorce.


(Gottman & Levenson, 2000)

Imbalance: f(betrayer) - f(victim)

(Im)balance Over Time

slide-66
SLIDE 66


 (Error bars show standard error.) positive sentiment

4 and up 3 2 1 betrayal −0.10 −0.05 0.00 0.05 0.10

politeness future planning time until betrayal imbalance

4 and up 3 2 1 betrayal −0.10 −0.05 0.00 0.05 0.10 4 and up 3 2 1 betrayal −0.10 −0.05 0.00 0.05 0.10

Imbalance: f(betrayer) - f(victim)

As betrayal 
 draws nearer, 
 balance is broken. Attributes change
 at different rates.

slide-67
SLIDE 67


 (Error bars show standard error.) positive sentiment

4 and up 3 2 1 betrayal −0.10 −0.05 0.00 0.05 0.10

politeness future planning time until betrayal imbalance

4 and up 3 2 1 betrayal −0.10 −0.05 0.00 0.05 0.10 4 and up 3 2 1 betrayal −0.10 −0.05 0.00 0.05 0.10

Imbalance: f(betrayer) - f(victim)

As betrayal 
 draws nearer, 
 balance is broken. Attributes change
 at different rates. Are these cues predictive?

slide-68
SLIDE 68

Predicting Betrayal

“I am supporting you into Sweden this turn if you want. If you want to be able to keep Sweden I suggest moving into Finland. 
 Cheers, Harriet Jones, PM.” “Thanks, I accept the support. I'll decide what I want to do with the army.” “Would it be ok with you if I took Denmark? I think I'm going to need it if I am going to hold France back.” “Hi Germany, How about I give you back Denmark next year. This is because I probably won't get a centre this year and would rather not disband a unit.”

slide-69
SLIDE 69

Predicting Betrayal

“Would it be ok with you if I took Denmark? I think I'm going to need it if I am going to hold France back.” “Hi Germany, How about I give you back Denmark next year. This is because I probably won't get a centre this year and would rather not disband a unit.”

Germany
 Stabs!

“Germany,
 Well that move was sour. This was a pity. Unfortunately now you have jumped out of the pan into the fire.”

slide-70
SLIDE 70

Toss in a few more features:

  • Sentiment

(Stanford Sentiment Analysis)
  • Argumentation & discourse

(Penn Discourse Treebank)
 (Stab & Gurevich, 2014)
  • Politeness 

(http://politeness.mpi-sws.org)
  • Subjectivity

(Riloff & Wiebe, 2003)
  • Talkativeness

Predicting Betrayal

slide-71
SLIDE 71

Predicting Betrayal

vs.

4 and up 3 2 1 betrayal −0.10 −0.05 0.00 0.05 0.10

Prediction tasks:

  • Will this friendship break?

(1375 seasons, 48% betrayals)
 Accuracy: (players: 52%)
 MCC:* (players: 0)
  • Is betrayal imminent?
slide-72
SLIDE 72

Predicting Betrayal

vs.

4 and up 3 2 1 betrayal −0.10 −0.05 0.00 0.05 0.10

Prediction tasks:

  • Will this friendship break?

(1375 seasons, 48% betrayals)
 Accuracy: (players: 52%)
 MCC:* (players: 0)
  • Is betrayal imminent?

(663 seasons from betrayals,
 14% immediately before betrayal)
 F1: (players: 0)
 MCC:* (players: 0)

*Matthews Correlation Coefficient: 0 = uninformative, 1 = perfect correlation.

slide-73
SLIDE 73

Predicting Betrayal

vs.

4 and up 3 2 1 betrayal −0.10 −0.05 0.00 0.05 0.10

Prediction tasks:

  • Will this friendship break?

(1375 seasons, 48% betrayals)
 Accuracy: (players: 52%) 57%
 MCC:* (players: 0) 0.14
  • Is betrayal imminent?

(663 seasons from betrayals,
 14% immediately before betrayal)
 F1: (players: 0)
 MCC:* (players: 0)

*Matthews Correlation Coefficient: 0 = uninformative, 1 = perfect correlation.

slide-74
SLIDE 74

Predicting Betrayal

vs.

4 and up 3 2 1 betrayal −0.10 −0.05 0.00 0.05 0.10

Prediction tasks:

  • Will this friendship break?

(1375 seasons, 48% betrayals)
 Accuracy: (players: 52%) 57%
 MCC:* (players: 0) 0.14
  • Is betrayal imminent?

(663 seasons from betrayals,
 14% immediately before betrayal)
 F1: (players: 0) 0.31
 MCC:* (players: 0) 0.17

*Matthews Correlation Coefficient: 0 = uninformative, 1 = perfect correlation.

slide-75
SLIDE 75

Predicting Betrayal

vs.

4 and up 3 2 1 betrayal −0.10 −0.05 0.00 0.05 0.10

Prediction tasks:

  • Will this friendship break?

(1375 seasons, 48% betrayals)
 Accuracy: (players: 52%) 57%
 MCC:* (players: 0) 0.14
  • Is betrayal imminent?

(663 seasons from betrayals,
 14% immediately before betrayal)
 F1: (players: 0) 0.31
 MCC:* (players: 0) 0.17
  • Outperforming the players!

*Matthews Correlation Coefficient: 0 = uninformative, 1 = perfect correlation.

slide-76
SLIDE 76

The intention to betray
 can leak through words. Good friendships
 are balanced. Imbalance changes
 as betrayal draws near.

0.05 0.15 0.25 positive sentiment betrayer victim 0.60 0.65 0.70 politeness

4 and up 3 2 1

0.05 0.10 0.15 planning positive sentiment −0.01 0.00 0.01 0.02 0.03 | imbalance | politeness planning matched friendship betrayal
slide-77
SLIDE 77

extra slides

slide-78
SLIDE 78 Positive sentiment I will still be trilled if you win this war. Negative sentiment It’s not a great outcome, but still an OK one. Neutral sentiment Do you concur with my assumption? Claim I believe that E/F have discarded him. Premise I put italy out because I wanted to work with you. Comparison We can trade centers as much as we like. Contingency He did not, thus we are indeed in fine shape. Expansion Would you rather see A or B? Temporal i think he can still be effective while you take ROM. Planning HOL should fall next year, and then MUN after. Subjectivity I’m just curious what you think. Politeness I wonder if you shouldn’t try to support Italy into MAR… What do you think?

Feature Examples

slide-79
SLIDE 79

Will this friendship break? Is betrayal imminent?

Sender Positive feature Sender Negative 
 feature B Positive sentiment B Expansion B
  • No. Sents
B Comparison B Contingency B
  • No. Words
B Planning B Negative sentiment Sender Positive feature Sender Negative 
 feature V Comparison B Claims V Positive sentiment B Politeness V Contingency B Contingency V Planning B Subjectivity V Requests B Expansion V Expansion B No. Sentences B Comparison

Selected Features