1 of 29

Semantic Search of Memes

on Twitter

Magdalena Saldaña

Jesús Pérez-Martín

Benjamin Bustos

2 of 29

Meme

Originally coined by the evolutionary biologist R. Dawkins as the corresponding cultural unit to the biological gene.

Ideas that are passed from host to carrier in a similar manner to genes, replicating, mutating and occasionally going extinct

"idea of a unit of cultural transmission, or a unit of imitation"

Richard Dawkins

3 of 29

Internet Meme

A virtually transmitted cultural symbol or social idea, that is shared on social media and is spread rapidly by Internet users.

“Internet Memes” can adapt to all formats available on the Internet (photo, video, text or audio)

"ideas that spread through populations via the Web. They can present as videos, Twitter hashtags, and cat photos with misspelled captions"

Steve Kolowich

"Internet meme" is a concept that rapidly gains popularity on the Internet and the research community. It is based on the concept of “meme” credited to Richard Dawkins since 1976

In general, this concept represents virtually transmitted cultural symbols or social ideas that are shared on social media and rapidly spread by Internet users.

“Internet Memes” can adapt to all formats available on the Internet, e.g., photo, video, text or audio.

Pronunciation

Internet -> ínternet

Rapidly -> rápidli

(con Meme slide) On the other hand, an "Internet meme" is another concept that rapidly gains popularity or notoriety on the Internet and the research community.

Since an increasing amount of social networks users are interacting through Memes, there is a growing need to quickly find the right Meme for a given message interchange, i.e., a user query.

Lo que borre>

They have become a constant of the online experience. While memes are most frequently shared on Facebook and Twitter, the sharing and replication of memes is not limited to these platforms.

They are also spread via email, WhatsApp, Facebook Messenger, Instagram, Snapchat, forums, and many other online platforms.

And there are a number of more obscure platforms such as 4Chan and Reddit on which many memes are generated. More on this later.

More concretely,

4 of 29

Context

Memes are an interesting mode of representation in the digital age.
In the fields of Computer Vision and Multimedia Information Retrieval, Internet memes have attracted the attention of the research community
Their adequate classification and retrieval, as opposed to general images, are challenging problems since they have a more complex semantic meaning that is often ironic and ambiguous.
Chile is a country with high rates of internet penetration and social media usage.

"We are living in a digital world that attaches deep value to the capacity to share, ant to laugh together at almost anything"

Tom Chatfield

Currently, “Internet Memes” are an interesting mode of representation in the digital age, which represents a new way of communication, and therefore they entail new and challenging tasks.

Specifically, in the fields of Computer Vision and Multimedia Information Retrieval, Internet Memes have attracted the attention of the research community as a new form of media that it is impossible to ignore.

In this study, we observe meme usage on Twitter in Chile, a country with high rates of Internet penetration and social media usage

Memes can adapt to all formats available on the Internet (photo, video, text, audio)

in the fields of Computer Vision and Multimedia Information Retrieval, Internet Memes have attracted the attention of the research community as new form of media that it is impossible to ignore.

In this study, we observe meme usage on Twitter in Chile, a country with high rates of internet penetration and social media usage

5 of 29

Motivation

Our long-term goal is to develop algorithms that allow us to determine which images are based on memes (or all the memes that have been created with the same image), for observing the discourse associated with said memes.

6 of 29

Objectives

To build a system that identifies Chilean accounts and download all tweets that have images.
To develop an algorithm capable of automatically classifying images in the meme, sticker, and no-meme classes.
To develop an algorithm able to rank and match a set of memes given a sentence as a query.

7 of 29

Preliminar Concepts

Classification

Semantic Search

Vector Model

Visual Descriptor

Text Embedding

8 of 29

SemanticMemes Dataset

9 of 29

SemanticMemes Dataset

Large dataset of textually described memes in Spanish.
Four experts manually classified 52,000 images (13,000 each one) from Chilean tweets posted from May to July, 2019.
Three classes: 1,194 Memes, 1,443 Stickers, and 49,347 No-memes.
The experts provided three textual annotations of each meme:
the text, a visual description, and semantic information
Spanish Vocabulary: 8,669 words.

Dataset website: https://jssprz.github.io/semantic-memes-docs/dataset/

This dataset is composed of a large amount of textually described memes in Spanish.

Four experts manually classified a total of 52 thousand images, 13 thousand each one, all obtained from Chilean tweets posted from May to July 2019.

They classified the images in three categories: one thousand one hundred ninety-four images as Memes, one thousand four hundred forty-three as Stickers, and the rest as No-memes.

Also, the experts generated three textual annotations of each meme.

The first one is a literal transcription of the text on the meme.

The second one is what we called the visual description of the contents in the meme, excluding the text.

And the third one refers to semantic information.

In total, the expert’s annotation uses eight thousand six hundred sixty-nine different words in Spanish.

10 of 29

SemanticMemes Dataset

Annotations:

Text in Meme: La política, explicada en una foto. nueva mayoría. tú, derecha. Alguna duda?
Visual Description: tres personas de espalda, sentadas en una banca. La persona de la izquierda y la derecha se están dando la mano por atrás de la banca.
Semantic Information: alude que las coaliciones políticas en chile se encuentran confabuladas y se ponen de acuerdo respecto a como funciona el pais, dejando de lado a la ciudadanía.

Here we can see an example of a meme and its related annotations.

For the Visual Description, which describes the visual contents in the photo. In this case, the expert wrote: “Three people from the back, sitting on a bench. The person on the left and the right are shaking hands behind the bench”.

****hasta akí****

As I mentioned before, the first annotation is a transcription of the text in the meme.

In this case (I’ll translate): “politics, explained in a photo. New majority, You, Right, Any doubt?”

The second annotation is the Visual Description, which describes the visual contents in the photo. In this case, the expert wrote: “Three people from the back, sitting on a bench. The person on the left and the right are shaking hands behind the bench”.

The last one is the Semantic Information, which is an interpretation of the meme. In this case, the translation will be: “It alludes that the political coalitions in Chile are conspiring and agree on how the country works, leaving aside the citizens”.

11 of 29

Why using Twitter?

Ideal for spreading news, with presence of traditional media.
For situations with little information, like rumors and spam, the first information appear on social networks, particularly on Twitter.
The people share their story on this social networks, and this information is public and free.
Machine Learning techniques allows us to extract opinions and other characteristics from the text of tweets.

12 of 29

Meme Recognition Task

13 of 29

Meme Recognition Task

Classify any image among the classes meme, sticker and no-meme.

Extract global visual representation of the image by ResNet-152 model
Train a SVM machine learning classifier.

14 of 29

Meme Recognition Task

Classify any image among the classes meme, sticker and no-meme.

Extract global visual representation of the image by ResNet-152 model
Train a SVM machine learning classifier.

15 of 29

Meme Recognition Task

Classify any image among the classes meme, sticker and no-meme.

Extract global visual representation of the image by ResNet-152 model
Train a SVM machine learning classifier.

16 of 29

Meme Retrieval Task

17 of 29

Challenges retrieving Internet Memes

More search criteria that general images, e.g., the text in the meme, the context in which was created, and in some cases, the contents of more than one photo.
More complex semantic meanings beyond their visual appearance.
The text of the memes is often ironic and thus captions that contain common words may express different ideas.
General image retrieval algorithm cannot be used effectively for the meme retrieval task.

But, this task is more complicated than the previous one. Then, before describing our method we need to mention some current challenges of the task.

First, compared to retrieving general images, the memes have more complex search criteria that general images, e.g., the text in the meme, the context in which it was created, and in some cases, the contents of more than one photo. Also, the relevance of these criteria may vary.

Memes have more complex semantic meanings beyond their visual appearance.

The associated text of the memes is often ironic, and thus captions that contain common words may express different ideas.

In our paper, the experiments show that a general image retrieval algorithm cannot be used effectively for the meme retrieval task.

For these limitations, this problem is increasing the attention of researchers of different communities.

18 of 29

Meme Retrieval Task

Learn a common visual-semantic-embedding.

A language model maps the texts to a language representation vector (text embedding).
A visual model produces a visual representation vector with semantic information.
Both vectors are projected into a shared feature space. Minimizing the distance between them.

Demo: https://s06.imfd.cl/04/twitter/demo/

19 of 29

Meme Retrieval Task

Learn a common visual-semantic-embedding.

A language model maps the texts to a language representation vector (text embedding).
A visual model produces a visual representation vector with semantic information.
Both vectors are projected into a shared feature space. Minimizing the distance between them.

Demo: https://s06.imfd.cl/04/twitter/demo/

20 of 29

Meme Retrieval Task

Learn a common visual-semantic-embedding.

A language model maps the texts to a language representation vector (text embedding).
A visual model produces a visual representation vector with semantic information.
Both vectors are projected into a shared feature space. Minimizing the distance between them.

Demo: https://s06.imfd.cl/04/twitter/demo/

21 of 29

Meme Retrieval Task

Learn a common visual-semantic-embedding.

A language model maps the texts to a language representation vector (text embedding).
A visual model produces a visual representation vector with semantic information.
Both vectors are projected into a shared feature space. Minimizing the distance between them.

Demo: https://s06.imfd.cl/04/twitter/demo/

22 of 29

Experimental Evaluation

Code: https://github.com/jssprz/SemanticMemes

23 of 29

Meme Recognition Experiments

SmanticMemes dataset distribution

Code: https://github.com/jssprz/SemanticMemes

24 of 29

Meme Recognition Experiments

SmanticMemes dataset distribution

Code: https://github.com/jssprz/SemanticMemes

25 of 29

Meme Recognition Results

Average results of Meme / Sticker / No-Meme classification on the Twitter Memes

Method	Precision	Recall	F1-Score
HOG + Naive Bayes	0.556	0.554	0.547
HOG + KNN	0.554	0.55	0.546
HOG + Linear-SVM	0.574	0.574	0.57
ResNet + Naive Bayes	0.641	0.633	0.631
ResNet + KNN	0.7	0.676	0.672
ResNet + Linear-SVM	0.73	0.73	0.73

Code: https://github.com/jssprz/SemanticMemes

26 of 29

Meme Recognition Results

Average Confusion Matrix results of ResNet + KNN method

Code: https://github.com/jssprz/SemanticMemes

27 of 29

Meme Retrieval Results

SemanticMemes dataset splits:

Training set: 6,228 pairs
Validation set: 289 pairs
Test set: 83 pairs

Hyper-parameters:

Epochs: 270
Batch-size: 16
Optimizer: Stochastic Gradient Descent (SGD)
Learning-rate: 0.0001
Criterion: Triplet Ranking Loss (margin=0.1)

Code: https://github.com/jssprz/SemanticMemes

28 of 29

Meme Retrieval Results

Training process

Demo: https://s06.imfd.cl/04/twitter/demo/

29 of 29

Thanks!!

jesus.perez@ug.uchile.cl

@jes_prz

Welcome to chatbox of the Advancement in Visual Analysis Paper Session.