Document Modeling with Gated Recurrent Neural Network for Sentiment - - PowerPoint PPT Presentation

Document Modeling with Gated Recurrent Neural Network for Sentiment Classification

Duyu Tang, Bing Qin, Ting Liu Harbin Institute of Technology

1

Sentiment Classification

• Given a piece of text, sentiment classification focus on inferring the

sentiment polarity of the text.

• Positive / Negative
• 1-5 stars
• The task can be at
• Word/phrase level, sentence level, document level
• We target at document-level sentiment classification in this work

2

Standard Supervised Learning Pipeline

Training Data Learning Algorithm Feature Representation Sentiment Classifier

3

Feature Learning Pipeline

Training Data Learning Algorithm Feature Representation Sentiment Classifier

Learn text representation/feature from data!

4

Deep Learning Pipeline

Training Data Learning Algorithm Feature Representation Sentiment Classifier Word Representation Words Semantic Composition

w1 w2 …… wn−1 wn

5

• Represent each word

as a low dimensional, real-valued vector

• Solutions: Word2Vec,

Glove, SSWE

Deep Learning Pipeline

Training Data Learning Algorithm Feature Representation Sentiment Classifier Word Representation Words Semantic Composition

w1 w2 …… wn−1 wn

6

• Compositionality: the meaning of a longer

expression depends on the meaning of its constituents

• Solutions at sentence level
• Recurrent NN, Recursive NN,

Convolutional NN, Tree-Structured LSTM

• Represent each word

as a low dimensional, real-valued vector

• Solutions: Word2Vec,

Glove, SSWE

The idea of this work

• We want to build an end-to-end neural network approach for

document level sentiment classification

• Human beings solve this problem in a hierarchical way: represent

sentence from words, and then represent document from sentences

• We want to use the semantic/discourse relatedness between

sentences to obtain the document representation

• We do not want to use an external discourse parser.

7

w1

1

w2

1

w3

1 w𝑚1−1 1

w𝑚1

1

Word Representation

w1

2

w2

2

w3

2 w𝑚2−1 2

w𝑚2

2

w1

𝑜

w2

𝑜

w3

𝑜 w𝑚𝑜−1 𝑜

w𝑚𝑜

𝑜

……

8

w1

1

w2

1

w3

1 w𝑚1−1 1

w𝑚1

1

CNN/LSTM

Word Representation Sentence Representation Sentence Composition

w1

2

w2

2

w3

2 w𝑚2−1 2

w𝑚2

2

CNN/LSTM w1

𝑜

w2

𝑜

w3

𝑜 w𝑚𝑜−1 𝑜

w𝑚𝑜

𝑜

CNN/LSTM

…… ……

9

w1

1

w2

1

w3

1 w𝑚1−1 1

w𝑚1

1

CNN/LSTM

Word Representation Sentence Representation Document Composition Sentence Composition

w1

2

w2

2

w3

2 w𝑚2−1 2

w𝑚2

2

CNN/LSTM w1

𝑜

w2

𝑜

w3

𝑜 w𝑚𝑜−1 𝑜

w𝑚𝑜

𝑜

CNN/LSTM Forward Gated Neural Network Forward Gated Neural Network Forward Gated Neural Network

…… ……

10

w1

1

w2

1

w3

1 w𝑚1−1 1

w𝑚1

1

CNN/LSTM

Word Representation Sentence Representation Document Composition Sentence Composition

w1

2

w2

2

w3

2 w𝑚2−1 2

w𝑚2

2

CNN/LSTM w1

𝑜

w2

𝑜

w3

𝑜 w𝑚𝑜−1 𝑜

w𝑚𝑜

𝑜

CNN/LSTM Forward Gated Neural Network Backward Gated Neural Network Forward Gated Neural Network Backward Gated Neural Network Forward Gated Neural Network Backward Gated Neural Network

…… ……

11

w1

1

w2

1

w3

1 w𝑚1−1 1

w𝑚1

1

CNN/LSTM

Word Representation Sentence Representation Document Composition Sentence Composition

w1

2

w2

2

w3

2 w𝑚2−1 2

w𝑚2

2

CNN/LSTM w1

𝑜

w2

𝑜

w3

𝑜 w𝑚𝑜−1 𝑜

w𝑚𝑜

𝑜

CNN/LSTM Forward Gated Neural Network Backward Gated Neural Network Forward Gated Neural Network Backward Gated Neural Network Forward Gated Neural Network Backward Gated Neural Network

…… ……

12

w1

1

w2

1

w3

1 w𝑚1−1 1

w𝑚1

1

CNN/LSTM

Word Representation Sentence Representation Document Representation Document Composition Sentence Composition

w1

2

w2

2

w3

2 w𝑚2−1 2

w𝑚2

2

CNN/LSTM w1

𝑜

w2

𝑜

w3

𝑜 w𝑚𝑜−1 𝑜

w𝑚𝑜

𝑜

CNN/LSTM Softmax Forward Gated Neural Network Backward Gated Neural Network Forward Gated Neural Network Backward Gated Neural Network Forward Gated Neural Network Backward Gated Neural Network

…… ……

13

14

Sentence Modeling

15

Sentence Modeling Document Modeling

Yelp 2015 (5-class) IMDB (10-class) Majority 0.369 0.179 SVM + Unigrams 0.611 0.399

16

Yelp 2015 (5-class) IMDB (10-class) Majority 0.369 0.179 SVM + Unigrams 0.611 0.399 SVM + Bigrams 0.624 0.409

17

Yelp 2015 (5-class) IMDB (10-class) Majority 0.369 0.179 SVM + Unigrams 0.611 0.399 SVM + Bigrams 0.624 0.409 SVM + TextFeatures 0.624 0.405

18

Yelp 2015 (5-class) IMDB (10-class) Majority 0.369 0.179 SVM + Unigrams 0.611 0.399 SVM + Bigrams 0.624 0.409 SVM + TextFeatures 0.624 0.405 SVM + AverageWordVec 0.568 0.319

19

Yelp 2015 (5-class) IMDB (10-class) Majority 0.369 0.179 SVM + Unigrams 0.611 0.399 SVM + Bigrams 0.624 0.409 SVM + TextFeatures 0.624 0.405 SVM + AverageWordVec 0.568 0.319 Conv-Gated NN (BiDirectional Gated Avg) 0.660 0.425

20

Yelp 2015 (5-class) IMDB (10-class) Majority 0.369 0.179 SVM + Unigrams 0.611 0.399 SVM + Bigrams 0.624 0.409 SVM + TextFeatures 0.624 0.405 SVM + AverageWordVec 0.568 0.319 Conv-Gated NN (BiDirectional Gated Avg) 0.660 0.425 LSTM-Gated NN 0.676 0.453

21

Yelp 2015 (5-class) IMDB (10-class) Majority 0.369 0.179 SVM + Unigrams 0.611 0.399 SVM + Bigrams 0.624 0.409 SVM + TextFeatures 0.624 0.405 SVM + AverageWordVec 0.568 0.319 Conv-Gated NN (BiDirectional Gated Avg) 0.660 0.425 Document Modeling Yelp 2015 (5-class) IMDB (10-class) Average 0.614 0.366 Recurrent 0.383 0.176

22

Yelp 2015 (5-class) IMDB (10-class) Majority 0.369 0.179 SVM + Unigrams 0.611 0.399 SVM + Bigrams 0.624 0.409 SVM + TextFeatures 0.624 0.405 SVM + AverageWordVec 0.568 0.319 Conv-Gated NN (BiDirectional Gated Avg) 0.660 0.425 Document Modeling Yelp 2015 (5-class) IMDB (10-class) Average 0.614 0.366 Recurrent 0.383 0.176 Recurrent Avg 0.597 0.344

23

Yelp 2015 (5-class) IMDB (10-class) Majority 0.369 0.179 SVM + Unigrams 0.611 0.399 SVM + Bigrams 0.624 0.409 SVM + TextFeatures 0.624 0.405 SVM + AverageWordVec 0.568 0.319 Conv-Gated NN (BiDirectional Gated Avg) 0.660 0.425 Document Modeling Yelp 2015 (5-class) IMDB (10-class) Average 0.614 0.366 Recurrent 0.383 0.176 Recurrent Avg 0.597 0.344 Gated NN 0.651 0.430

24

Yelp 2015 (5-class) IMDB (10-class) Majority 0.369 0.179 SVM + Unigrams 0.611 0.399 SVM + Bigrams 0.624 0.409 SVM + TextFeatures 0.624 0.405 SVM + AverageWordVec 0.568 0.319 Conv-Gated NN (BiDirectional Gated Avg) 0.660 0.425 Document Modeling Yelp 2015 (5-class) IMDB (10-class) Average 0.614 0.366 Recurrent 0.383 0.176 Recurrent Avg 0.597 0.344 Gated NN 0.651 0.430 Gated NN Avg 0.657 0.416

25

In Summary

• We develop a neural network approach for document level sentiment

classification.

• We model document with gated recurrent neural network, and we

show that adding neural gates could significantly boost the classification accuracy.

• The codes and datasets are available at: http://ir.hit.edu.cn/~dytang

26

Thanks

27