SentiBERT: A Transferable Transformer-based Architecture for - PowerPoint PPT Presentation

SentiBERT: A Transferable Transformer-based Architecture for Compositional Sentiment Semantics Da Yin 1 , Tao Meng 2 , Kai-Wei Chang 2 1 Peking University 2 University of California, Los Angeles 1

Motivation ● Sentiment composition is challenging. positive . neutral negative Frenetic but not really funny Frenetic but not really funny. 2

Motivation ● How to encode sentiment composition in a contextual encoder? ● Can semantic composition learned from SST transfer to related tasks? Better capture = + sentiment composition 3

Model Sentiment Phrase Node BERT Semantics Prediction Composition 4

Model BERT Sentiment Semantics Composition ● Layer 1: Attention to Tokens Phrase Node ● Layer 2: Attention to Children Prediction 5

Training Objectives 6

Experiments ● Tasks: ○ SST-phrase Evaluated under supervised learning ○ SST-5 protocol ○ SST-2, SST-3 ○ Twitter Sentiment Analysis Test transferability ○ Contextual Emotion Detection (EmoContext) ○ Emotion Intensity Classification (EmoInt) 7

Experiments ● Results: ○ For sentiment semantic composition: SST-phrase (Accuracy) SST-5 (Accuracy) 0.7 0.58 0.57 0.69 0.56 0.68 0.55 0.67 0.54 0.53 0.66 0.52 0.65 0.51 0.5 0.64 0.49 0.63 0.48 0.62 0.47 BERT BERT w/ Tree-LSTM BERT BERT w/ Tree-LSTM SentiBERT RoBERTa SentiBERT RoBERTa SentiBERT w/ RoBERTa SentiBERT w/ RoBERTa More results and discussion are in the paper 8

Experiments ● Results: ○ For transferability: EmoInt (Pearson Correlation) EmoContext (F1) 0.675 0.75 0.67 0.745 0.665 0.74 0.66 0.655 0.735 0.65 0.73 0.645 0.64 0.725 BERT SentiBERT BERT SentiBERT RoBERTa SentiBERT w/ RoBERTa RoBERTa SentiBERT w/ RoBERTa More results and discussion are in the paper 9

Analysis -- Performance v.s. Sentiment Switch positive . neutral negative Frenetic but not really funny ● Local difficulty: the number of sentiment switches between a phrase and its children ● Global difficulty: the number of sentiment switches in the entire constituency tree 10

Analysis ● Results: Local Difficulty Global Difficulty More results and discussion are in the paper 11

Case Study More examples are in the paper 12

Conclusion ● We present SentiBERT to better capture compositional sentiment semantics ● SentiBERT can transfer the compositional sentiment semantics learned on SST to other related tasks Thanks! GitHub: https://github.com/WadeYin9712/SentiBERT 13