Content Models with Attitude Christina Sauper, Aria Haghighi, Regina - - PowerPoint PPT Presentation

Content Models with Attitude

Christina Sauper, Aria Haghighi, Regina Barzilay MIT

1

Review Aggregation

• Hundreds of reviews for each product
• Opinions vary widely

→ Need aggregate statistics

• Histograms show sentiment distribution, but it’s not enough

2

Aspect-based Analysis

Prior work:

Use a set of predefined domain-specific product aspects

(e.g., Snyder and Barzilay 2007)

→ Coarse level analysis

3

Informative Aggregation

Useful information:

– What’s the best dish at this restaurant? – What do people dislike about this restaurant? – Which dishes do people disagree about?

4

We had a great time last night at this restaurant. The sushi was so incredibly fresh. We had a bad experience at the bar, though. My chocolate martini was absolutely terrible. We will be back, but we’ll skip the drinks. Wow, I can’t believe how much this place has changed! They used to be mediocre, but now they never fail to amaze. We started off at the bar with awesome sake

• bombs. When we got to
• ur table, the sushi was

fantastic. I have such mixed things to say about this

• restaurant. On one

hand, their sushi is unquestionably the best in the city. On the other, the atmosphere isn’t that great. Plus, their drinks are completely watered down.

Aggregation of product-specific aspects

Informative Aggregation

We had a great time last night at this restaurant. The sushi was so incredibly fresh. We had a bad experience at the bar, though. My chocolate martini was absolutely terrible. We will be back, but we’ll skip the drinks. Wow, I can’t believe how much this place has changed! They used to be mediocre, but now they never fail to amaze. We started off at the bar with awesome sake

• bombs. When we got to
• ur table, the sushi was

fantastic. I have such mixed things to say about this

• restaurant. On one

hand, their sushi is unquestionably the best in the city. On the other, the atmosphere isn’t that great. Plus, their drinks are completely watered down.

Sushi Chicken 100% positive 33% positive

Japanese Restaurant

Relevant aspects User sentiment

5

Corpus-driven Aspect Definition

Define aspects dynamically based on reviews

We had a great time last night at this

• restaurant. The sushi

was so incredibly fresh. We had a bad experience at the bar,

• though. My chocolate

martini was absolutely

• terrible. We will be

back, but we’ll skip the drinks. Wow, I can’t believe how much this place has changed! They used to be mediocre, but now they never fail to amaze. We started

• ff at the bar with

awesome sake bombs. When we got to our table, the sushi was fantastic. I have such mixed things to say about this

• restaurant. On one

hand, their sushi is unquestionably the best in the city. On the

• ther, the atmosphere

isn’t that great. Plus, their drinks are completely watered down.

Bakery

• Cookies
• Cakes
• Pies

We had a great time last night at this

• restaurant. The sushi

was so incredibly fresh. We had a bad experience at the bar,

• though. My chocolate

martini was absolutely

• terrible. We will be

back, but we’ll skip the drinks. Wow, I can’t believe how much this place has changed! They used to be mediocre, but now they never fail to amaze. We started

• ff at the bar with

awesome sake bombs. When we got to our table, the sushi was fantastic. I have such mixed things to say about this

• restaurant. On one

hand, their sushi is unquestionably the best in the city. On the

• ther, the atmosphere

isn’t that great. Plus, their drinks are completely watered down.

Japanese Restaurant

• Sushi
• Sake
• Dessert

→ Aspects specific to each product

6

Corpus-driven Aspect Definition

Allows comparison across multiple reviews

– Consensus (both positive and negative)

What’s the best/worst aspect of this product?

I buy all of my baked goods from this

• bakery. Their bread is

so delicious! It’s also good for all kinds of baked goods. They also have some truly beautiful cakes

• n
• display. Even their

cookies are great! I picked up a birthday cake for my son here

• yesterday. It was the

most amazing cake I’ve ever seen! The decorations were

• utstanding, and all

the kids loved the chocolate icing. I’ll definitely come back! This place is nice for some baked goods, but some things are really nasty. The loaf

• f bread I bought was

stale! They were happy to take it back and give me another, but I’ll be watching next time.

Bakery

…truly beautiful cakes on display. …most amazing cake I’ve ever seen!

7

Corpus-driven Aspect Definition

Allows comparison across multiple reviews

– Consensus (both positive and negative)

What’s the best/worst aspect of this product?

– Conflicts of opinion

What aspects do people disagree about?

I buy all of my baked goods from this

• bakery. Their bread is

so delicious! It’s also good for all kinds of baked goods. They also have some truly beautiful cakes

• n
• display. Even their

cookies are great! I picked up a birthday cake for my son here

• yesterday. It was the

most amazing cake I’ve ever seen! The decorations were

• utstanding, and all

the kids loved the chocolate icing. I’ll definitely come back! This place is nice for some baked goods, but some things are really nasty. The loaf

• f bread I bought was

stale! They were happy to take it back and give me another, but I’ll be watching next time.

Bakery

Their bread is so delicious! The loaf of bread I bought was stale!

8

Task: Input

Input:

– Food-related snippets from restaurant reviews

• Concise description of a user’s opinion

– Automatically extracted from full review text (Sauper et al. 2010) – Segmented by restaurant, but no additional annotation

the sushi was so incredibly fresh best chicken katsu in town drinks are fun, fresh, and delicious I’d recommend the apple pie the bread was disappointingly stale chocolate torte is the stuff of dreams 9

Japanese Restaurant Bakery

We went to the restaurant, and the sushi was incredibly fresh.

Task: Output

Output:

– Relevant aspects for each restaurant – Aspect label for each snippet – Sentiment label for each snippet

10 + they had a decent burrito − the burrito was mediocre at best − the burrito was heavily cilantroed + the salsa is incredible + the mango salsa is perfectly diced + hola free chips & salsa Burrito Salsa Mexican Restaurant

Possible Solution

Use clustering based on lexical similarity Problem: Clusters and aspects are not aligned!

the martinis were very good the martinis were tasty the wine list was pricey their wine selection is horrible the sushi was the best I’d ever had best paella I’d ever had the fillet was the best steak we’d ever had it’s the best soup I’ve ever had

Partial output of state-of-the-art clustering system

11

Our Solution

• Jointly model aspect and sentiment
• Leverage data to distinguish relevant words

12

Bakery Japanese Review 1 Review 2 Review 3

delicious fresh fantastic amazing beautiful stale fantastic smooth beautiful fresh delicious bland pies cookies cakes pies cakes bread salmon sake maki salmon maki miso

Model: Overview

• Each snippet has an aspect and a sentiment
• Each word is drawn from a topic distribution:

– Aspects are specific to a single product – Sentiment is global across all products – Background distribution is global

• Transition distribution encodes word topic transitions

great horrible amazing dessert pizza pad thai

• ur

was food

13

They had wonderful appetizers.

Model: Generative Story

• 1. Global distributions
• 2. Restaurant-level distributions
• 3. Snippet-level latent structure
• 4. Words

14

Model: Generative Story

B

Background distribution Sentiment distributions

+

• Globally,
• a. Background distribution

word distribution for stop words and in-domain white noise

• b. Sentiment distributions ,

word distributions over positive and negative sentiment words small bias for seed words

• c. Transition distribution

first-order Markov distribution of word topic transitions

Λ

Transition distribution

15

Model: Generative Story

For each restaurant ,

• a. Aspect distributions

word distribution for each aspect

• b. Aspect-sentiment binomials

probability of positive vs. negative sentiment for each aspect

• c. Aspect multinomial

probability of each aspect

Aspect distributions

1 … 2 K ψ

Aspect multinomial Aspect-sentiment binomials

… φ1 φ2 φK

16

Model: Generative Story

For each snippet ,

• a. Aspect

chosen from aspect multinomial

• b. Sentiment

chosen from aspect-sentiment binomial

• c. Sequence of word topics

Background, Aspect, or Sentiment selected from transition distribution

2 ψ

Aspect

φ2 +

Sentiment Word topic sequence

B B A S S Λ

17

Model: Generative Story

For each word ,

• a. Word

chosen from topic-specific distribution based on word topic sequence

2 +

Aspect Sentiment Word topic sequence

B B A S S

Background

B B B A S S

The pizza was really great

2 + B

18

Standard Variational Inference

• Desired posterior:

Model parameters Observed data Latent structure

19

Standard Variational Inference

• Desired posterior:
• Optimizing directly is intractable
• Instead, optimize variational objective:

20

s.t. factorizes

Standard Variational Inference

• Mean-field factorization:
• Optimization strategy:

minimize each while holding others constant (coordinate descent)

Background Sentiments Aspects Latent variables Products

21

Standard Variational Inference

Updates:

– Parameter updates are straightforward via latent variable counts – Latent variable updates are derived from expectations

E.g.: update for probability that snippet aspect equals 22

Expected log probability of

• Exp. log prob. of aspect words in the snippet coming from
• Exp. log prob. that has the chosen sentiment

Data Set

Food-related snippets from Yelp restaurant reviews

(Sauper et al. 2010)

– 13,879 total snippets – 328 restaurants – 42.1 snippets per restaurant (high variance) – 7.8 words per snippet

Seed words for sentiment distributions

– 42 positive, 33 negative – Relevant to domain (e.g., “delicious”)

23

Experiments: Aspect Clustering

• Gold standard

– Clusters over 3,250 snippets – Collected via Mechanical Turk

• Baseline

– CLUTO clustering weighted by TF*IDF

• MUC cluster evaluation metric

– Based on number of cluster merges and splits required to achieve gold data

• Both systems allowed 10 clusters per restaurant

24

Experiments: Aspect Clustering

69.3 75.5 60 70 80 Baseline Our model

MUC F1

the martinis are very good the martini selection looked delicious the s’mores martini sounded excellent Our model the martinis are very good the mozzarella was very fresh the fish and various meets were well made Baseline Baseline the carrot cake was delicious it was rich, creamy, and delicious the pasta bolognese was rich and robust Our model the carrot cake was delicious the best carrot cake I’ve ever eaten carrot cake was deliciously moist

25

Experiments: Sentiment Analysis

• Gold standard

– 260 snippets (130 train / 130 test) – Manually labeled POSITIVE or NEGATIVE

• Baselines

– DISCRIMINATIVE: binary classifier over unigrams – SEED: positive and negative seed word counts

26

Experiments: Sentiment Analysis

75.9 78.2 80.2 70 75 80 85 DISCRIMINATIVE SEED OUR MODEL

Accuracy

Our model

27

Error Analysis

• Goal:

– Identify common errors

• Procedure:

– Select 102 clusters – Manually annotate correctness of aspect and sentiment

28

Error Analysis

Number of sentiment and aspect errors approximately equal

29

Aspect errors − Similar aspect words in different contexts Sentiment errors − Rare sentiment words − Negation, sometimes

the cream cheese was n’t bad belgian frites are very crave-able the blackened chicken was meh chicken enchiladas are yummy the cream cheese wasn’t bad ice cream was just delicious

Related Work

• Single-review analysis

Hu and Liu 2004; Popescu et al. 2005; Kim and Hovy 2006

– Focus on single reviews

• Multi-document summarization

Liu et al. 2005b; Carenini et al. 2006; Hu and Liu 2006; Kim and Zhai 2009

– Also present contrastive viewpoints and average sentiment – Focus on extracting relevant sentences

• Probabilistic topic models

Mei et al. 2007; Lu and Zhai 2008; Titov and McDonald 2008

– Focus on coarser document-level sentiment

30

Conclusion

• Aspect and sentiment can be modeled jointly for

review aggregation

• Inference can be done efficiently for this model using

variational techniques

• Joint model boosts the performance of both aspect

selection and sentiment analysis

31