Multimodal Language Analysis in the Wild: CMU-MOSEI Dataset and - PowerPoint PPT Presentation

Multimodal Language Analysis in the Wild: CMU-MOSEI Dataset and Interpretable Dynamic Fusion Graph Presenter: Paul Pu Liang Amir Zadeh, Paul Pu Liang, Jonathan Vanbriessen, Soujanya Poria, Edmund Tong, Erik Cambria, Minghai Chen, Louis-Philippe Morency 1 Paul Pu Liang Multimodal Language Analysis in the Wild: CMU-MOSEI Dataset and Interpretable Dynamic Fusion Graph

Progress of Artificial Intelligence Intelligent Robots and Multimedia Content Personal Assistants Virtual Agents 2 Paul Pu Liang Multimodal Language Analysis in the Wild: CMU-MOSEI Dataset and Interpretable Dynamic Fusion Graph

Continuous Theories of (Multimodal) Language Throughout evolution language and nonverbal behaviors developed together. Cries and Imitations Modern Language Paul Pu Liang Multimodal Language Analysis in the Wild: CMU-MOSEI Dataset and Interpretable Dynamic Fusion Graph

Multimodal Language Modalities Language Visual Ø Lexicon Ø Gestures Ø Syntax Ø Body language Ø Pragmatics Ø Eye contact Ø Facial expressions Acoustic Ø Prosody Ø Vocal expressions 4 Paul Pu Liang Multimodal Language Analysis in the Wild: CMU-MOSEI Dataset and Interpretable Dynamic Fusion Graph

Multimodal Language Modalities Language Visual Sentiment Ø Positive Ø Lexicon Ø Gestures Ø Negative Ø Syntax Ø Body language Emotion Ø Anger Ø Pragmatics Ø Eye contact Ø Disgust Ø Facial expressions Ø Fear Ø Happiness Acoustic Ø Sadness Ø Surprise Ø Prosody Personality Ø Vocal expressions Ø Confidence Ø Persuasion Ø Passion 5 Paul Pu Liang Multimodal Language Analysis in the Wild: CMU-MOSEI Dataset and Interpretable Dynamic Fusion Graph

Multimodal Language Modalities Sentiment Language Visual Acoustic Emotion Personality Datasets Models 6 Paul Pu Liang Multimodal Language Analysis in the Wild: CMU-MOSEI Dataset and Interpretable Dynamic Fusion Graph

Multimodal Language Modalities Sentiment Language Visual Acoustic Emotion Personality Datasets Models 7 Paul Pu Liang Multimodal Language Analysis in the Wild: CMU-MOSEI Dataset and Interpretable Dynamic Fusion Graph

Multimodal Language Modalities Sentiment Language Visual Acoustic Emotion Personality Datasets Models ü Large-scale ü Diverse 8 Paul Pu Liang Multimodal Language Analysis in the Wild: CMU-MOSEI Dataset and Interpretable Dynamic Fusion Graph

Multimodal Language Modalities Sentiment Language Visual Acoustic Emotion Personality Datasets Models § Word-level alignment § Attention models § Memory-based models ü Large-scale ü Diverse 9 Paul Pu Liang Multimodal Language Analysis in the Wild: CMU-MOSEI Dataset and Interpretable Dynamic Fusion Graph

Multimodal Language Modalities Sentiment Language Visual Acoustic Emotion Personality Datasets Models § Word-level alignment § Attention models § Memory-based models ü Large-scale ü Good Performance ü Diverse ü Interpretable 10 Paul Pu Liang Multimodal Language Analysis in the Wild: CMU-MOSEI Dataset and Interpretable Dynamic Fusion Graph

Datasets for Multimodal Language § Require large and diverse amounts of data: § Diversity in samples Paul Pu Liang Multimodal Language Analysis in the Wild: CMU-MOSEI Dataset and Interpretable Dynamic Fusion Graph

Datasets for Multimodal Language § Require large and diverse amounts of data: § Diversity in samples § Diversity in topics Paul Pu Liang Multimodal Language Analysis in the Wild: CMU-MOSEI Dataset and Interpretable Dynamic Fusion Graph

Datasets for Multimodal Language § Require large and diverse amounts of data: § Diversity in samples § Diversity in topics § Diversity in speakers Paul Pu Liang Multimodal Language Analysis in the Wild: CMU-MOSEI Dataset and Interpretable Dynamic Fusion Graph

Datasets for Multimodal Language § Require large and diverse amounts of data: § Diversity in samples § Diversity in topics § Diversity in speakers § Diversity in annotations Paul Pu Liang Multimodal Language Analysis in the Wild: CMU-MOSEI Dataset and Interpretable Dynamic Fusion Graph

New Dataset: CMU-MOSEI 23,000 video segments 3 modalities 15 Paul Pu Liang Multimodal Language Analysis in the Wild: CMU-MOSEI Dataset and Interpretable Dynamic Fusion Graph

CMU-MOSEI Dataset 1,000 speakers 250 topics 16 Paul Pu Liang Multimodal Language Analysis in the Wild: CMU-MOSEI Dataset and Interpretable Dynamic Fusion Graph

Annotation Distributions 17 Paul Pu Liang Multimodal Language Analysis in the Wild: CMU-MOSEI Dataset and Interpretable Dynamic Fusion Graph

Annotation Distributions 18 Paul Pu Liang Multimodal Language Analysis in the Wild: CMU-MOSEI Dataset and Interpretable Dynamic Fusion Graph

Feature Extraction Language Sentiment Ø Positive Ø Glove word embeddings Ø Negative Visual Emotion Alignment Ø Anger Ø Facet features Ø Disgust Ø Word level Ø MultiComp OpenFace Ø Fear Ø P2FA Ø Face embeddings Ø Happiness Acoustic Ø Sadness Ø Surprise Ø COVAREP features MFCCs • Pitch tracking • 19 Paul Pu Liang Multimodal Language Analysis in the Wild: CMU-MOSEI Dataset and Interpretable Dynamic Fusion Graph

CMU-MOSEI Dataset Multimodal Language Audio-visual 20 Paul Pu Liang Multimodal Language Analysis in the Wild: CMU-MOSEI Dataset and Interpretable Dynamic Fusion Graph

Models for Multimodal Language ! ! ! ! multimodal Multimodal Fusion Paul Pu Liang Multimodal Language Analysis in the Wild: CMU-MOSEI Dataset and Interpretable Dynamic Fusion Graph

Models for Multimodal Language Interpretation ! multimodal § Importance of each modality § Interactions between modalities Multimodal Fusion

Dynamic Fusion Graph (DFG) Interpretation $ multimodal § Importance of each modality § Interactions between modalities unimodal " " ! ! # # Paul Pu Liang Multimodal Language Analysis in the Wild: CMU-MOSEI Dataset and Interpretable Dynamic Fusion Graph

Dynamic Fusion Graph (DFG) Interpretation $ multimodal § Importance of each modality § Interactions between modalities bimodal unimodal " " ! ! # # Paul Pu Liang Multimodal Language Analysis in the Wild: CMU-MOSEI Dataset and Interpretable Dynamic Fusion Graph

Dynamic Fusion Graph (DFG) Interpretation $ multimodal § Importance of each modality § Interactions between modalities trimodal bimodal unimodal " " ! ! # # Paul Pu Liang Multimodal Language Analysis in the Wild: CMU-MOSEI Dataset and Interpretable Dynamic Fusion Graph

Dynamic Fusion Graph (DFG) Interpretation $ multimodal § Importance of each modality § ⊕ Interactions between modalities fusion weights trimodal bimodal unimodal " ! # Paul Pu Liang Multimodal Language Analysis in the Wild: CMU-MOSEI Dataset and Interpretable Dynamic Fusion Graph

Dynamic Fusion Graph (DFG) Interpretation $ multimodal § Importance of each modality § ⊕ Interactions between modalities fusion weights trimodal bimodal unimodal " ! # Paul Pu Liang Multimodal Language Analysis in the Wild: CMU-MOSEI Dataset and Interpretable Dynamic Fusion Graph

Dynamic Fusion Graph (DFG) t = 1 $ multimodal ⊕ trimodal bimodal unimodal " ! # Paul Pu Liang Multimodal Language Analysis in the Wild: CMU-MOSEI Dataset and Interpretable Dynamic Fusion Graph

Dynamic Fusion Graph (DFG) t = 1 t = 2 $ multimodal ⊕ trimodal bimodal unimodal " ! # Paul Pu Liang Multimodal Language Analysis in the Wild: CMU-MOSEI Dataset and Interpretable Dynamic Fusion Graph

Dynamic Fusion Graph (DFG) t = 1 t = 2 t = 3 $ multimodal ⊕ trimodal bimodal unimodal " ! # Paul Pu Liang Multimodal Language Analysis in the Wild: CMU-MOSEI Dataset and Interpretable Dynamic Fusion Graph

Dynamic Fusion Graph (DFG) t = 1 t = 2 t = 3 t = 4 $ multimodal ⊕ trimodal bimodal unimodal " ! # Paul Pu Liang Multimodal Language Analysis in the Wild: CMU-MOSEI Dataset and Interpretable Dynamic Fusion Graph

Dynamic Fusion Graph (DFG) Interpretation Interpretation $ multimodal § Importance of each modality § Interactions between modalities fusion weights trimodal bimodal unimodal " ! # Paul Pu Liang Multimodal Language Analysis in the Wild: CMU-MOSEI Dataset and Interpretable Dynamic Fusion Graph

Dynamic Fusion Graph (DFG) Interpretation $ multimodal § Importance of each modality § Interactions between modalities fusion weights trimodal § Construction of bimodal and trimodal representations bimodal construction weights unimodal " ! # Paul Pu Liang Multimodal Language Analysis in the Wild: CMU-MOSEI Dataset and Interpretable Dynamic Fusion Graph

Dynamic Fusion Graph (DFG) Interpretation $ multimodal § Importance of each modality § Interactions between modalities fusion weights trimodal § Construction of bimodal and trimodal representations bimodal construction weights ! ",$ ! %,$ ! ",% unimodal ' & ( Paul Pu Liang Multimodal Language Analysis in the Wild: CMU-MOSEI Dataset and Interpretable Dynamic Fusion Graph