Words Can Shift: Dynamically Adjusting Word Representations using - - PowerPoint PPT Presentation

Words Can Shift: Dynamically Adjusting Word Representations using Nonverbal Behaviors

Yansen Wang, Ying Shen, Zhun Liu, Paul Pu Liang, Amir Zadeh, Louis-Philippe Morency

“This is unbelievable!”

Speaker’s behaviors time

Multimodal Sentiment Analysis

Non-verbal behaviors influence the meaning of words

Smile Loud

Dynamically Shifting in Word Representation Space

Dynamically Shifting in Word Representation Space

Dynamically Shifting in Word Representation Space

Modeling nonverbal behaviors

movie

𝑏"

($)

𝑏&

($)

𝑏'()

($)

𝑀𝑇𝑈𝑁/ 𝑀𝑇𝑈𝑁/ 𝑀𝑇𝑈𝑁/

⋯ Acoustic Visual ℎ2

($)

ℎ/

($)

Visual Embedding Acoustic Embedding

sick

t = 2.8

…

time

𝑀𝑇𝑈𝑁2 𝑀𝑇𝑈𝑁2

𝑤"

($)

𝑀𝑇𝑈𝑁2

⋯ 𝑤&

($)

𝑤'4)

($)

Nonverbal Sub-networks

last night.

⋯ ⋯ ⋯ ⋯ t = 2.3

Modality Gating

movie

𝑏"

($)

𝑏&

($)

𝑏'()

($)

𝑀𝑇𝑈𝑁/ 𝑀𝑇𝑈𝑁/ 𝑀𝑇𝑈𝑁/

⋯ Acoustic Visual ℎ2

($)

ℎ/

($)

𝑓($)

Visual Embedding Acoustic Embedding Word Embedding

Attention Gating sick

t = 2.8

…

time

Gated Modality-mixing Network

𝑀𝑇𝑈𝑁2 𝑀𝑇𝑈𝑁2

𝑤"

($)

𝑀𝑇𝑈𝑁2

⋯ 𝑤&

($)

𝑤'4)

($)

Nonverbal Sub-networks

last night.

⋯ ⋯ ⋯ ⋯ t = 2.3

Nonverbal Shift

ℎ6

($)

Dynamic shifts

movie

𝑏"

($)

𝑏&

($)

𝑏'()

($)

𝑀𝑇𝑈𝑁/ 𝑀𝑇𝑈𝑁/ 𝑀𝑇𝑈𝑁/

⋯ Acoustic Visual ℎ2

($)

ℎ/

($)

𝑓($)

Visual Embedding Acoustic Embedding Word Embedding Nonverbal Shift

Attention Gating

𝑓6

($)

sick

t = 2.8

Shifting …

time

Gated Modality-mixing Network

𝑓($) sick’ 𝑓6

($)

ℎ6

($)

Shifting

ℎ6

($)

Multimodal Shifting

𝑀𝑇𝑈𝑁2 𝑀𝑇𝑈𝑁2

𝑤"

($)

𝑀𝑇𝑈𝑁2

⋯ 𝑤&

($)

𝑤'4)

($)

Nonverbal Sub-networks

Multimodal-shifted word representation

last night.

⋯ ⋯ ⋯ ⋯ t = 2.3

sick

Please come to our poster Thursday 6:30-8:30pm @ Coral, South Pacific and Nautilus Lounges

Thank you!

http://multicomp.cs.cmu.edu/