​

​

​

​

​

​

Twitter Image and Text Data Analysis for Preserving Ethical Practices in Social Media

​

Presented by

Devtanu Misra (510519009)

Anish De (510519010)

Ahel Das Chatterjee (510519028)

Malay Gain (510519033)

Sounak Majumder (510519068)

Sayak Rana (510519108)

​

Under guidance of

Prof. Jaya Sil

​

2

Contents

3

Introduction

• Social media platforms have become essentials of our daily routine and twitter is one of them.
• So, we wanted to analyse the twitter posts and find whether any negative sentiment is carried by it.
• We used Multimodal(image & text), Multilingual(English+ Hindi) approach.
• We divided a post in four components namely Post caption, Comments, Image Caption, Image OCR to get full idea of a post.
• We created Sentweet, an entirely fresh dataset by ourselves.
• We used transformer models(Deberta, XLM-Roberta, Bert Multilingual,Electra ) for training purpose.

4

Motivation/Objective

• With huge increase in use of social medias, unsocial activities also increased in these platforms.
• We need some method to identify whether some post is derogatory or not.
• There are some models in place but they focus on text or image separately.
• Our model works in a multimodal multilingual fashion to give more emphasis on capturing the underlying sentiment.
• Also our newly created dataset sentweet gives us the idea of modern posts from different sectors.
• We have used pretrained models which have been already trained with some benchmark datasets.

5

Related Work

• Ullah MA, Syeda M, Begum SA, Dipa NS. An algorithm and method for sentiment analysis using the text and emoticons from social media data. This study was done using machine learning and deep learning algorithms, both of which performed well.
• Zhang Y, Sun J, Meng L, Liu Y. Sentiment analysis of e-commerce text reviews based on sentiment dictionary. Our work is similar to this, to the extent of using different parts of speech.
• Jitendra V. Tembhurne1 & Tausif Diwan1. Sentiment analysis in textual, visual and multimodal inputs using recurrent neural networks.
• Aria Naseri Karimvand, Reza Salehi Chegeni, Mohammad Ehsan Basiri , Shahla Nemati . Sentiment Analysis of Persian Instagram Post: a Multimodal Deep Learning Approach.
• Sani Kamış , Dionysis Goularas. Evaluation of Deep Learning Techniques in Sentiment Analysis from Twitter Data.

6

Proposed Work

• Data Scraping - Scrape data from twitter to prepare a dataset consisting of multilingual posts - posts in Hindi, English and Bengali
• Dataset Creation - Consider 4 categories to classify the posts - Profane, Targeted Insult, Sentiment and Hate Speech
• Manually determine the classes for each post
• Image Caption and Image OCR - Use the post image to extract text OCR as well image caption, and push them into our dataset
• Training our model - Use the newly created dataset to train our models and compare their accuracy
• Predict unknown data - Use our model to categorize the nature of unknown data, and label them as unethical, vulgar or offensive

7

Proposed Work - Flowchart

8

Twitter APIs

Get comments -

"https://api.twitter.com/2/tweets/search/recent?query=conversation_id:1295468508697026562

&tweet.fields=in_reply_to_user_id,author_id,created_at,conversation_id"

​

Get media files -

"https://api.twitter.com/2/tweets?ids=1579762797462032386&expansions=attachments.media_keys

&media.fields=duration_ms,height,media_key,preview_image_url,public_metrics,type,url,width,alt_text"

​

Get conversation id -

"https://api.twitter.com/2/tweets?ids=1225917697675886593&

tweet.fields=author_id,conversation_id,created_at,in_reply_to_user_id,referenced_tweets&

expansions=author_id,in_reply_to_user_id,referenced_tweets.id&user.fields=name,username"

9

Twitter APIs - Flowchart

10

Dataset Preparation

Here’s how we prepared our dataset:

• We scrapped the data using Twitter APIs provided by Twitter Developer Account.
• Pushed twitter post texts, corresponding comments under the post, their media url and language of the post under a single row.
• Manually classified the posts under 4 headings: Hate Speech, Profane, Sentiment and Targeted Insult
• Images downloaded in a single folder using the media url extracted in previous step
• Algorithms and models for extracting image OCR and image captions were run on the files in this folder
• These text inserted into the excel file to increase volume of data.
• Around 6000 data points in our dataset, roughly 4500 from original post texts and around 1500 extracted as image OCR and captions

11

Dataset Preparation

Here’s how we prepared our dataset:

• POST_ID, POST LINK : Received as response from Twitter APIs
• Text : The actual post texts, along with hashtags, emojis, etc
• Text Type : Whether the scrapped text is a Caption, Comment, Image OCR or Image Caption.
• MEDIA_URL : Endpoints that store the media posted alongside this post
• Hate Speech : Whether the post incites riots, hate between communities, etc. Might be TRUE or FALSE
• Profane : Whether the post has vulgar or obscene remarks. Might be TRUE or FALSE
• Sentiment : Kind of emotion the post raises in a person. Might Positive, Negative or Neutral
• Targeted Insult : Whether the post is directed as an insult to a particular group or people. Might be YES or NO

12

Image Preprocessing

• Image Preprocessing has been done before performing OCR to make as easy as possible for the OCR system to distinguish a character/word from the background.
• Preprocessing Techniques include-
• Binarization- This method gives a threshold for the whole image considering the various characteristics of the whole image (like lighting conditions, contrast, sharpness etc).
• Skew Correction- We can correct the skewness (image aligned at a certain angle with horizontal) by rotating the image.
• Noise Removal- The main objective of the Noise removal stage is to smoothen the image by removing small dots/patches which have high intensity than the rest of the image.

13

Image OCR

• Pytesseract tool has been used for converting image-text to text.
• It can read all image types including jpeg, png, gif, bmp, tiff, and others.
• It has unicode (UTF-8) support, and can recognize more than 100 languages "out of the box".

Tesseract-Architecture

14

• Pytesseract is actually a wrapper class or a package for Google’s Tesseract-OCR Engine.
• Tesseract OCR includes a new neural network set which is based on LSTM neural network.
• It is focused on Word, Line character recognition.

Image OCR Algorithm

• For this, at-first we are importing the required libraries.
• Then we read the images column-wise from the combined csv and import images one-by-one from the images folder prepared and pre-process them.
• Later we are storing the generated OCR text from the images column-wise in the csv in a separate column.

15

Sample Image Input:

Sample OCR Output from Model:

16

Image Captioning

• Image Captioning is the process of generating textual description of an image.

Image Captioning -Architecture

• Approach-1

17

• In the Approach-1 we are basically training a model using the Flickr dataset which consists of around 8000 images and their respective captions.
• After that we are testing with the images extracted from twitter posts and generating respective captions.

Sample Image Input:

Sample Output from Model:

a man in a white dog is standing on the water

18

• Approach-2

• In the Approach-2 we are basically using a pre-trained (already trained on some dataset) Image Captioning Model, namely ViT-Roberta model as a pipeline.
• After that we are testing it with the images extracted from twitter posts and generating respective captions.

19

Sample Image Input:

Sample Output from Model:

a man in a suit with a microphone

• Lastly what we can do is we can download a pre-trained model, fine-tune (train & evaluate) the model using Flickr dataset and use this as a pipeline.

• After importing the pipeline, we read the images column-wise from the combined csv and import images one-by-one from the images folder prepared.
• Later we are storing the generated captions from the images column-wise in the csv in a separate column.

20

Transformers

​

A transformer is a deep learning model that adopts the mechanism of self-attention, differentially weighting the significance of each part of the input data.

Advantages of Transformers over CNN

• CNNs are pretty good at feature extraction. CNN does not encode the relative position of different features.
• Large filters are required to encode the combination of these features which require large filters. Large receptive fields are required in order to track long-range dependencies within an image.
• Increasing the size of the convolution kernels can increase the representational capacity of the network but doing so also loses the computational and statistical efficiency obtained by using local convolutional structure.
• The self-attention mechanism is a type of attention mechanism which allows every element of a sequence to interact with every others and find out who they should pay more attention to.

​

21

BERT Model

​

​

BERT is basically an Encoder stack of transformer architecture. A transformer architecture is an encoder-decoder network that uses self-attention on the encoder side and attention on the decoder side.

22

Roberta Model

It essentially includes fine-tuning the original BERT model along with data and inputs manipulation.

In many ways this is a better version of the BERT model. The key points of difference are as follows:

a. Dynamic Masking: BERT uses static masking i.e. the same part of the sentence is masked in each Epoch. In contrast, RoBERTa uses dynamic masking, wherein for different Epochs different part of the sentences are masked. This makes the model more robust.

b. Remove NSP Task: It was observed that the NSP task is not very useful for pre-training the BERT model. Therefore, the RoBERTa only with the MLM task.

c. More data Points: BERT is pre-trained on “Toronto BookCorpus” and “English Wikipedia datasets” i.e. as a total of 16 GB of data. In contrast, in addition to these two datasets, RoBERTa was also trained on other datasets like CC-News (Common Crawl-News), Open WebText etc. The total size of these datasets is around 160 GB.

​

23

Transformers vs LSTM

• Non-sequential - Transformers process sentences as whole.
• Self attention - Used to find similarity scores between words in a sentence.
• Brings to power of bidirectional RNNs to get past and future data.

24

Experimental Data - Sentiment

Table: Experimental Data showing Accuracy, F1, precision and recall for sentiment model

25

AOC ROC & Confusion Matrix - Sentiment

26

Experimental Data - Profane

Table: Experimental Data showing Accuracy, F1, precision and recall for profane model

27

Experimental Data - Hate Speech

Table: Experimental Data showing Accuracy, F1, precision and recall for hate speech model

28

Experimental Data - Targeted Insult

Table: Experimental Data showing Accuracy, F1, precision and recall for targeted insult model

29

Conclusion

According to the work done till now, we found -

-> Most of the texts are non-hatred.

-> Accuracy can be improved using a larger size dataset.

-> Accuracy can also be improved using a balanced dataset.

-> Image captioning can be improved using more images and training for long hours.

-> Looking forward to incorporate all these things in the next semester.

​

30

Future Work

• Increase in the number of models to further improve the evaluation metrics to an extent.
• Now we are generating 4 different sentiments(i.e Hate Speech, Profane, Sentiment and Targeted Insult) about a post using 4 different models. Later, we will be merging these 4 models to get a final output about the post whether it is derogatory or not.
• Increase the dataset by introducing automation to improve training.
• Further improvisation is required related to OCR and Image captionings.
• Learn and understand and decode the sentiment associated with emojis.

​

31

References

• https://medium.com/@kalpeshmulye/image-captioning-using-hugging-face-vision-encoder-decoder-step-2-step-guide-part-1-495ecb05f0d5

​

• https://huggingface.co/blog/sentiment-analysis-python

​

• https://nanonets.com/blog/ocr-with-tesseract/

​

• https://medium.com/@raman.shinde15/image-captioning-with-flickr8k-dataset-bleu-4bcba0b52926

​

32