MemeSequencer : Sparse Matching for Embedding Image Macros Abhimanyu - - PowerPoint PPT Presentation

MemeSequencer : Sparse Matching for Embedding Image Macros

Abhimanyu (Abhi) Dubey, Esteban Moro, Manuel Cebrian and Iyad Rahwan

Massachusetts Institute of Technology {dubeya*,emoro,cebrian,irahwan}@mit.edu

Image Virality

• Thousands of images are shared online every day
• Predicting virality can benefit:
• content creation (designing content that becomes viral)
• content-based network traffic routing (routing content based on predicted virality)
• content-based network caching (caching the content that is likely to go viral)
• There has been incredible prior work in the area of predicting viral content based on

network structure of information dissemination (cascades, community detection etc.)

• However, there has been limited work on the impact of the content itself on its virality
• In this study we focus solely on the content and its impacts on popularity

MemeSequencer | Abhimanyu Dubey (WWW 2018)

Memes as Images

MemeSequencer | Abhimanyu Dubey (WWW 2018)

Image Macro

Memes evolve during propagation

MemeSequencer | Abhimanyu Dubey (WWW 2018)

(a) (b) (c) (d)

Types of mutations in memes

MemeSequencer | Abhimanyu Dubey (WWW 2018)

(b) (c) (a) Content-Preserving Content Altering

Content-preserving mutations (overlays)

MemeSequencer | Abhimanyu Dubey (WWW 2018)

Designing an embedding for memes

MemeSequencer | Abhimanyu Dubey (WWW 2018)

d ( , ) d ( , ) d ( , ) d ( , )

<

Rn

Key methodological insight

MemeSequencer | Abhimanyu Dubey (WWW 2018)

What if we could recover the

• riginal image (template) from the

modified image? If we can decouple the mutations from the original image, we can create a space that preserves the required distances.

Decoupled semantic embeddings

MemeSequencer | Abhimanyu Dubey (WWW 2018)

template information mutation information

We create an augmented set by creating affine transformations of the templates: Consider the set of k template images:

Sparse Matching

MemeSequencer | Abhimanyu Dubey (WWW 2018)

...

Each target image y can then be represented (with error) as a linear combination

• f the template images si,j (with αi,j as the weights):

Sparse Matching

MemeSequencer | Abhimanyu Dubey (WWW 2018)

In matrix form, we can replace the above by the equation:

We then want to recover x0 such that:

Sparse Matching

MemeSequencer | Abhimanyu Dubey (WWW 2018)

In the current form, however, this problem is NP-hard (even to approximate), so we solve the following problem:

The ultimate goal of the method is to create a semantic embedding that enables us to understand virality. The steps of the overall algorithm that provides us with a feature embedding are:

• Sparse Matching to decouple the template and mutations
• Image feature extraction on template and the mutation separately
• CNN-based feature extractors such as AlexNet, ResNet etc
• Text feature extraction on template and mutation separately
• Run off-the-shelf OCR (OpenCV) to identify text
• Extract generic text features (Average Word2Vec and Skip-Thought) to obtain semantic

information

• Use classifier (SVM) on features to make predictions based on task

Constructing decoupled feature vectors

MemeSequencer | Abhimanyu Dubey (WWW 2018)

• Typically we have the template set predefined (or manually constructed)
• How do we proceed in the case when we do not have a template set?
• We proceed without the template set, and construct it on the go
• We proceed with an empty template set
• For each image we encounter, we run the sparse matching algorithm
• If the image doesn’t match with an existing template, we add it to our template set
• If the image matches with an existing template, we update the template by median blending:
• For all images that match the template, we set the final template image as the median

(pixel-wise)

Constructing the template set

MemeSequencer | Abhimanyu Dubey (WWW 2018)

Median Blending

MemeSequencer | Abhimanyu Dubey (WWW 2018)

Overall pipeline

MemeSequencer | Abhimanyu Dubey (WWW 2018)

target macro set (T) template set (S) Sparse Matching

x 𝜷

matched template decoupled overlay OCR Image Feature Extractor (CNN) Text Feature Extractor (SkipThought or Word2Vec)

• h you just

graduated? you must know everything. Feature Representation template image features

• verlay image

features

• verlay text

features

Experimental Setup

MemeSequencer | Abhimanyu Dubey (WWW 2018)

We examine our algorithm on 3 different datasets:

• Viral Images Dataset (Parikh2015, Lakkaraju2013): 6k training, 500 test image pairs
• Memegenerator Dataset (Coscia2013): 326,181 images (70-10-20 train-val-test split)
• Quickmeme Dataset (Coscia2013): 178,801 images (70-10-20 train-val-test split)

For baselines, we use the following methods:

• Spatial Transformer Networks (Dubey2017)
• Low level vision features (Parikh2015)
• CNN based feature extractors (AlexNet, VGGNet, ResNet)
• Text based feature extractors (Word2Vec, SkipThought)

How separable is our representation?

MemeSequencer | Abhimanyu Dubey (WWW 2018)

• We extracted sparse matching features for all 500k images
• Clustered these images based on K-means with varying number of clusters
• Calculated Silhouette Score (SS)
• Measures how much intra-cluster dissimilar exists: lower is better for tight clusters
• Calculated Davies-Bouldin Index (DBI)
• Measures how similar different clusters are : lower is better for good separation

How separable is our representation?

MemeSequencer | Abhimanyu Dubey (WWW 2018)

Benefits in nearest neighbor retrieval

MemeSequencer | Abhimanyu Dubey (WWW 2018)

Benefits in pairwise virality prediction

MemeSequencer | Abhimanyu Dubey (WWW 2018)

• We extracted sparse matching features for all images (per dataset)
• For extracted features, we train a RankSVM to solve the following task:
• Given a pair of images, which image is likely to go viral?

Benefits in pairwise virality prediction

MemeSequencer | Abhimanyu Dubey (WWW 2018)

Uncovering mutation patterns via phylogenetic trees

MemeSequencer | Abhimanyu Dubey (WWW 2018)

Take-home summary

MemeSequencer | Abhimanyu Dubey (WWW 2018)

• We propose the study of image virality based on content alone
• Most memes and viral images are types of image macros - predefined image

templates with added images or text (overlays)

• We develop an algorithm that exploits this structure of macros to create a

robust semantic embedding, which provides benefits on

• Semantic clustering
• Image retrieval
• Virality, topic and popularity prediction
• Uncovering mutation patterns in memes