SLIDE 1
Opinion Spam Analysis and Detection
Leaked Confidential Information as Ground Truth
Yu-Ren Chen, Hsin Hsi Chen National Taiwan University
SLIDE 2 What Is Opinion Spam?
- spreading commercially advantageous opinions as
regular users on the Internet
○ positive opinions about own products/services ○ negative opinions about competitors
- also known as ‘review spam’ or ‘shilling’
- undoubtedly unethical
- in most cases illegal
- believed to be widely used in practice
○ potentially lucrative profits → strong incentives
SLIDE 3 ‘Spam’ in General
○ (Wikipedia) the use of electronic messaging systems to send unsolicited bulk messages (spam), especially advertising, indiscriminately ○ (Oxford) irrelevant or unsolicited messages sent
- ver the Internet, typically to large numbers of
users, for the purposes of advertising, phishing, spreading malware, etc.
SLIDE 4 Various Kinds of Spam
○ the most well-known kind of spam ○ spam is defined as ‘email spam’ in Merriam-Webster
○ manipulation of search engine indices and rankings ○ content spam: keyword stuffing, meta-tag stuffing, scraper sites, article spinning, machine translation ○ link spam: link farms, hidden links, expired domains, comment spam, referrer log spam
○ bulk message, malicious links, fake friends, etc.
SLIDE 5 Opinion Spam v.s. Other Spam
- very carefully written to avoid getting caught
○ deemed as fraud → unacceptable ○ backfiring may cause serious damage to the reputation of a brand (or a store, restaurant, etc.)
- initiated by a big brand in our case study
(Samsung)
○ high stakes ○ opinion spammers have to be really careful
SLIDE 6 Difficulty in Obtaining Dataset
- manual annotation is pretty much useless
○ low inter-annotator agreement score (Ott et al. 2011) ○ approximated ground truth ■ duplicate / near-duplicate reviews
■ crowdsourced fake reviews
- (Ott et al. 2011)
- utilizing confidential internal records in our study
○ real record of opinion spam in real world ○ ‘true’ ground truth
SLIDE 7 Our Case Study - 三星寫手門事件
- conducted by ‘鵬泰顧問有限公司’, a subsidiary
company of Samsung
- hired writers and designated employees were
instructed to make disingenuous reviews on web forums
- revealed by a hacker known as ‘0xb’
○ making confidential documents in 鵬泰 publicly available on TaiwanSamsungLeaks.org
- Samsung was fined $10 million TWD by the Fair Trade
Commission in Taiwan
SLIDE 8 About ‘Mobile01’
- the main battlefield of this marketing campaign
- where the leaked documents was first made public
- one of the most popular websites in Taiwan
○ #10 in Alexa traffic rank in Taiwan
- mainly featuring discussion about consumer electronics
○ mobile phones, tablets, etc.
- primarily a web forum site
○ rather than product review site such as Amazon
- written in Traditional Chinese for the most part
○ rather than in English
SLIDE 9
Structure of Mobile01 Forum
SLIDE 10 More about Web Forums
○ collection of posts (from oldest to latest) ○ started by specifying the title and the first post
○ first post (original post, thread starter, 一樓) ○ reply (回覆) ■ all posts except the first posts ■ can be used for bumping (手動置頂，頂)
○ forum ⇢ board ⇢ thread ⇢ post
SLIDE 11 Dataset Collection
○ HHP-2011.xlsx and HHP-2012.xlsx ○ containing URLs to the spam posts ○ source of the ground truth in our study
○ contents of the posts ■ including spam and non-spam ○
- ther various kinds of information on Mobile01
○ all posts from 2011 to 2012 on SAMSUNG (Android) board ■ where the ‘spam density’ is the highest ○ all profiles of the posters of such posts ○ three SQLite tables: POSTS, PROFILES, THREADS,
SLIDE 12
a snippet of the leaked spreadsheet ‘HHP-2012.xlsx’
SLIDE 13
an example post on Mobile01
SLIDE 14
a snippet of the table ‘POSTS’ scraped from Mobile01
SLIDE 15
an example profile on Mobile01
SLIDE 16
a snippet of the table ‘PROFILES’ scraped from Mobile01
SLIDE 17
a snippet of the table ‘THREADS’ scraped from Mobile01
SLIDE 18
amount of data collected
SLIDE 19 Looking into the Dataset
○ subtlety in spam posts ○ low spam post ratio of some spammers ○ types of spammer accounts
■ reputable accounts ■ throwaway accounts
○ first posts v.s. replies in threads ○ pattern in submission time of spam posts ○ activeness of threads ○ collusive activities of spammers
SLIDE 20
examples of subtly written spam posts
SLIDE 21
examples of subtly written spam posts
SLIDE 22
examples of subtly written spam posts
SLIDE 23 Low Spam Post Ratio of Some Spammers
- our definition of spammer
○ poster who had submitted any spam post
- only 33% of the posts from spammers are spam in this
dataset
SLIDE 24 Different Types of Spammer Accounts
○ hired reputable writers ○ low spam ratio
○ registered in batch ○ high spam ratio ○ low number of threads initiated
○ hired non-reputable posters? ○ borrowed accounts?
SLIDE 25
pattern in registration time
SLIDE 26 First Posts v.s. Replies
○ initiating the thread ○ richer in content ○ higher spam ratio
- reply (2nd, 3rd posts in thread)
○ usually quite concise
SLIDE 27 Submission Time of Spam Posts
- hypothesis: spam posts are more often made during
work time compared to normal posts
○ because spamming is a job rather than a leisure activity
SLIDE 28 Activeness of Threads
- threads started by spam first posts are expected to be
more active
○ written to draw attention and exposure
- measuring of ‘activeness’ of a thread
○ number of posts in the thread ○ number of clicks on the thread
SLIDE 29 Collusion between Spammers
- different spam accounts submit spam posts to the
same thread ○ fabricating the majority opinion ○ bumping the same spam thread
- 67% of the spam posts are in the threads containing
multiple spam posts
- could be different actual human posters
- or just one human login with different accounts
○ still can be seen as collusion between the accounts
SLIDE 30 Detection
- evaluation metric
- data splitting
- machine learning
- spam detection for first posts
- spam detection for replies
- spammer detection
SLIDE 31 Evaluation Metric
- spam / spammer is in minority (<5%)
○ accuracy ✘
- high precision / recall on the spam /
spammer class is preferable
○ F-measure ✔
SLIDE 32 Data Splitting
○ made in 2011 → training set ○ made between Jan 2012 and May 2012 → test set
- user accounts (spammer detection)
○ who had submitted a post in 2011 → training set ○ who had submitted a post between Jan 2012 and May 2012 → test set ○ who had submitted a post in both → training set ■ probabilistic prediction on posts will be used in spammer detection
SLIDE 33 Test Set* for Posts
- concern: capturing the writing habit of spammers?
○ favorite words, preferred writing style and etc. ■ might be purely personal preference ■ not essential to opinion spamming
- solution: removing posts by ‘cross-posters’ from test
set for posts ○ cross-posters: users who have made a post in both
- f training set and test set
○ result set: test set*
SLIDE 34
number of instances in each split
SLIDE 35 Machine Learning
- Scikit-Learn (Pedregosa et al. 2011)
○ machine learning in Python
○
- utperforms SVM with linear kernel, logistic regression,
AdaBoost, random forests, etc ○ Python wrapper for LibSVM ○ scaling features to zero mean and unit variance ■ (Hsu et al. 2003) ○ two primary hyperparameter (c, γ) to tune ■ 5-fold cross-validation on the training set ■ grid search on (c, γ) with F-measure as the metric to
SLIDE 36 Spam Detection for First Posts
- specifically for first posts in threads
- expected to be have nicer results than for replies
○ higher spam ratio ○ richer content
- only show the performance on test set* in the
following slides for conciseness
SLIDE 37 Random Baseline
- predict whether a first post is spam
according to the result of a fair coin flip
- precision ≈ ratio of spam
- recall ≈ 50%
features\metrics precision recall F-measure random 2.52% 55.71% 4.82%
SLIDE 38 (Dimension-reduced) Bag-of-Words
- Chinese word segmentation with Jieba
- words with < 5 occurrences are removed
- words appeared in over 30% of the posts are removed
○ stop words like
- Randomized PCA (Halko et al. 2011) to reduce the
dimension of bag-of-words
○ efficient on large matrices ○ mitigating overfitting ○ speed up the training process
SLIDE 39 Number of Dimension to Reduce to
- determined by the result of the 5-fold cross validation
- F-measure is the highest when bag-of-words is reduced
to 150 dimensions
SLIDE 40 Bag-of-Words Performance
- tremendous performance boost
○ F-measure improved by 46%
features\metrics precision recall F-measure random 2.52% 55.71% 4.82% bag-of-words 50.00% 51.43% 50.70%
SLIDE 41 Finding the ‘Keywords’
- finding the features that decide the predictions
○ no simple way since RBF kernel is non-linear
- falling back to SVM with linear kernel
○ suffering ~10% loss in F-measure ○ able to get the coefficients (weights) which indicate the importance of each feature
- PCA inverse-transformed the coefficients to get the
importance of each word
SLIDE 42
words with the highest weights (spam keywords)
SLIDE 43
words with the lowest weights (ham keywords)
SLIDE 44 Comparing the ‘Keywords’
- words that are the strongest spam indicators
○ Samsung’s top products (galaxy, nexus, note, sii) ○ describing user experiences (體驗 , 看到 , 覺得) ○ focusing on the multimedia aspect (照片 , 拍照 , 影片)
- words that are the strongest non-spam indicators
○ seeking help (問題, 解決, 無法) ○ polite words (謝謝, 大大, 小弟) ○ technicalities (rom, 開機 , 設定)
- spam first posts put more emphasis on certain topics
○ bag-of-words able to exploit such characteristics ○ although each post looks rather unsuspicious
SLIDE 45 Bag-of-words of Titles
- titles of the threads may help too
○ but might have a different set of keywords ○ appending the title bag-of-words reduced to 50 dimensions
features\metrics precision recall F-measure random 2.52% 55.71% 4.82% bag-of-words 50.00% 51.43% 50.70% title + content bag-of-words 56.16% 58.57% 57.34%
SLIDE 46
Content Characteristics
SLIDE 47
symmetric KL divergence between spam and hams
SLIDE 48
distribution of #images and #words
SLIDE 49 Content Quantity of Spam First Posts
- spam first posts are essentially advertisements in
disguise
○ ‘unboxing’ articles ○ experience with the products ○ containing lots of words and pictures
- regular users usually don’t bother to take effort to
write long articles
○
SLIDE 50 Impact on Performance
○ We prefer precision to be higher anyway.
- overall F-measure increased by 3%
features\metrics precision recall F-measure random 2.52% 55.71% 4.82% bag-of-words 56.16% 58.57% 57.34% bag-of-words + content CHAR 64.91% 52.86% 60.32%
SLIDE 51 Non-content-related Features
○ 24 binary indicator features for each hour ○ 7 binary indicator features for each day in a week
○ number of posts in the thread initiated by the first post
features \ metrics precision recall F-measure random 2.52% 55.71% 4.82% bag-of-words 56.16% 58.57% 57.34% BoWs + content CHAR 64.91% 52.86% 60.32% BoWs + content CHAR + time + thread len 66.67% 57.97% 62.02%
SLIDE 52 Moving on to Detection for Replies
- compared with first posts
○ lower ratio of spam ○ less content ○ expected to be harder to detect
- replies sharing the same thread with some replies in
training set are removed
SLIDE 53 Repeating the Procedures
○ worse F-measure due to lower spam ratio
○ not nearly as useful as for first posts
features\metrics precision recall F-measure random 1.38% 50.15% 2.68% bag-of-words 11.85% 19.83% 14.83% bag-of-words + content CHAR. 12.32% 19.83% 15.20%
SLIDE 54 Content Characteristics of Replies
- three out of the top four content characteristics
features in terms of symmetric KL divergence
○ n_emoticons, p_emoticon_pos, p_ntusd_neg ○ all about sentiment ○ side with the previous ‘pro-samsung’ posts
SLIDE 55 Content-centric Features Not as Effective for Replies
- replies tend to contain substantially less content that
first posts
○ don’t need to initiate a topic discussion ○ concise first posts are less acceptable in the many forum cultures
- replies that are ‘just for bumping’
○ drawing more attention to the thread ■
- ften started by a spam first post
○
- ften not even trying to deliver any message
■ already in other posts in the thread
SLIDE 56 Non-content-related Features
- previously used: submission time, length of thread
- an additional one: position of the reply in the thread
○ nfloor and pnum
features\metrics precision recall F-measure random 1.38% 50.15% 2.68% bag-of-words 11.85% 19.83% 14.83% bag-of-words + content CHAR. 12.32% 19.83% 15.20% BoWs + content
14.65% 25.07% 18.49%
SLIDE 57 ‘Spamicity’ of the First Post in Thread
- reply to a thread started by a spam first post
○ more likely to be spam
- spamicity of the first post in the thread
○ leveraging the model in spam detection for first posts ■ probabilistic prediction ■ more likely to be spam if closer to 1
features\metrics precision recall F-measure bag-of-words 11.85% 19.83% 14.83% BoWs + content
14.65% 25.07% 18.49% 〃+ 1st post spamicity 21.10% 26.82% 23.62%
SLIDE 58 Spammer Detection
○
poster who had submitted any spam post
- each instance in the training and test set is an user
account ○ instead of a post
- suitable for some applications
○ just ban the spammers
SLIDE 59 Features from User Profile
- numerical attributes from scraped profiles
○ n_threads, p_phone_section, score, n_posts, n_replies
- spammer accounts actually tend to be more ‘reputable’
○ hired reputable writers ○ higher degree of participation ○ most normal users are ‘lurkers’
features\metrics precision recall F-measure random 0.91% 45.88% 1.78% profiles 2.75% 22.62% 4.91%
SLIDE 60 Maximum Spamicity of First Posts of the User
- maximum spamicity of all first posts made by the user
○ taking the max is a natural choice considering our definition of spammer
- preferring the model for first posts to have
precision > recall
○ ‘multiple chances’ to get it right ○
- ur model for first post happens to have this property
- if an user in test set(users) have a spam first post in
training set(posts)
○ cheating?
SLIDE 61 Impact on Performance
- almost a 50% boost in F-measure
- 54 / 84 (64.3%) of the spammer in test set has a spam
in training set for first posts
features\metrics precision recall F-measure random 0.91% 45.88% 1.78% profiles 2.75% 22.62% 4.91% profiles + max_spamicity_fps 66.67% 42.86% 52.17%
SLIDE 62 Burstiness of Throwaway Spammer Account Registrations
- number of throwaway spammer accounts registered
within 20 days
features\metrics precision recall F-measure random 0.91% 45.88% 1.78% profiles 2.75% 22.62% 4.91% profiles + max_spamicity_fps 66.67% 42.86% 52.17% profiles + max_spamicity_fps + burstiness_reg 67.27% 44.05% 53.24%
SLIDE 63 Frequently Appeared Groups of Posters
- utilizing the ‘collusion between spammers’
- bservation mentioned earlier
- finding the groups of posters who often appear (make
post) together in the same threads ○ frequent itemset mining
■ each itemset is 30 consecutive posters in a thread ■ implementation: Python Orange
SLIDE 64 As a ‘Smoothing’ Step
- not directly incorporated as a feature into the model
- if the sum of spamicity of the posters in a frequent
appeared group is > threshold ○ predict all posters in that group as spammers
SLIDE 65
pseudocode of the remained parts
SLIDE 66 Impact on Performance
features\metrics precision recall F-measure random 0.91% 45.88% 1.78% profiles 2.75% 22.62% 4.91% profiles + max_spamicity_fp s 66.67% 42.86% 52.17% profiles + max_spamicity_fp s + burstiness_reg 67.27% 44.05% 53.24% profiles + max_spamicity_fp s + burstiness_reg + freq groups 67.95% 63.10% 65.43%
SLIDE 67 Caveat
- burstiness of throwaway spammer account
registrations
○ result is meaningless if the test set is not temporally contiguous to training set
- frequent appeared groups of posters
○ similar situation
SLIDE 68 Conclusions
- the first dataset with the “true ground truth”
for opinion spam studies
- content-centric features are ‘unintuitively’
effective for finding spam first posts
- spam replies seem to be much harder to
detect than first posts
- collusion between spammers is a big giveaway
SLIDE 69 Future Work
- sentiment shown in posts toward the brands
○ an algorithm to better capture sentiment of the posts
- interaction between posters
○ what forums are used for ○ reply to whom?
- integration of spam and spammer detection
○ just different perspectives ○ currently: first post → spammer ○ an united model?
SLIDE 70
Questions?
