Words Can Shift: Dynamically Adjusting Word Representations using - PowerPoint PPT Presentation

Words Can Shift: Dynamically Adjusting Word Representations using Nonverbal Behaviors Yansen Wang, Ying Shen, Zhun Liu, Paul Pu Liang, Amir Zadeh, Louis-Philippe Morency

Multimodal Sentiment Analysis Speaker’s behaviors “This is unbelievable!” Smile Loud time Non-verbal behaviors influence the meaning of words

Dynamically Shifting in Word Representation Space

Dynamically Shifting in Word Representation Space

Dynamically Shifting in Word Representation Space

Modeling nonverbal behaviors Nonverbal Sub-networks ⋯ 𝑀𝑇𝑈𝑁 2 𝑀𝑇𝑈𝑁 2 𝑀𝑇𝑈𝑁 2 Visual Embedding Visual ($) ℎ 2 ⋯ ($) ($) ($) 𝑤 ' 4) 𝑤 " 𝑤 & ⋯ ⋯ 𝑀𝑇𝑈𝑁 / 𝑀𝑇𝑈𝑁 / 𝑀𝑇𝑈𝑁 / Acoustic Acoustic Embedding ⋯ ($) ℎ / ($) ($) ($) 𝑏 " 𝑏 & 𝑏 ' () ⋯ t = 2.3 t = 2.8 time … movie last night. sick

Modality Gating Nonverbal Sub-networks Gated Modality-mixing Network ⋯ 𝑀𝑇𝑈𝑁 2 𝑀𝑇𝑈𝑁 2 𝑀𝑇𝑈𝑁 2 Visual Embedding Visual ($) Nonverbal ℎ 2 ⋯ Shift Word ($) Embedding ($) ($) Attention 𝑤 ' 4) 𝑤 " 𝑤 & ⋯ Gating ($) 𝑓 ($) ℎ 6 ⋯ 𝑀𝑇𝑈𝑁 / 𝑀𝑇𝑈𝑁 / 𝑀𝑇𝑈𝑁 / Acoustic Acoustic Embedding ⋯ ($) ℎ / ($) ($) ($) 𝑏 " 𝑏 & 𝑏 ' () ⋯ t = 2.3 t = 2.8 time … movie last night. sick

Dynamic shifts Nonverbal Sub-networks Gated Modality-mixing Multimodal Shifting Network ⋯ 𝑀𝑇𝑈𝑁 2 𝑀𝑇𝑈𝑁 2 𝑀𝑇𝑈𝑁 2 Visual Embedding Visual ($) Nonverbal Multimodal-shifted ℎ 2 ⋯ Shift word representation Word ($) Embedding ($) ($) Attention 𝑤 ' 4) 𝑤 " 𝑤 & ⋯ Shifting Gating ($) ($) 𝑓 ($) 𝑓 6 ℎ 6 ⋯ 𝑀𝑇𝑈𝑁 / 𝑀𝑇𝑈𝑁 / 𝑀𝑇𝑈𝑁 / Acoustic ($) sick’ ℎ 6 sick Acoustic ($) 𝑓 6 𝑓 ($) Embedding ⋯ ($) ℎ / ($) ($) ($) 𝑏 " 𝑏 & 𝑏 ' () ⋯ Shifting t = 2.3 t = 2.8 time … movie last night. sick

Thank you! Please come to our poster Thursday 6:30-8:30pm @ Coral, South Pacific and Nautilus Lounges http://multicomp.cs.cmu.edu/