Sarcasm Identification of Dravidian Languages (Malayalam and Tamil) in DravidianCodeMix @ FIRE 2024

13-12-2024

N Sripriya

Sri Sivasubramaniya Nadar College of Engineering, Chennai, Tamil Nadu

Sripriya N,

Bharathi B,

Thenmozhi Durairaj,

Nandhini K,

Rahul Ponnusamy,

Prasanna Kumar Kumaresan,

Kishore Kumar Ponnusamy and

Charmathi Rajkumar

Bharathi Raja Chakravarthi

AUTHORS

Introduction

Task Description

Dataset

Methodology

Results and Discussion

I

II

III

IV

V

Table of

Contents

VI

Conclusion

• Sarcasm is the use of words to express the opposite of what is meant, often to mock or ridicule someone/something.

​

Eg: When someone does wrong, when a person says well- done, very good, it is sarcastic

​

• The emergence of social media platforms like WhatsApp, Instagram, Twitter, and Facebook has led to the widespread of sarcastic content among the public.

​

• Sarcasm identification has become highly crucial due to its significance in related fields like sentiment analysis and emotion recognition.

​

• It could provide businesses and politicians with significant insight, as it reflects the opinion of people.

​

​

​

INTRODUCTION

I

INTRODUCTION

I

• Comments or posts on social media are often code-mixed.

​

• Sarcasm identification in code-mixed languages is a crucial task in natural language processing.

​

​

• This presentation provides an in-depth examination of the Sarcasm Identification shared task held as part of DravidianCodeMix@FIRE-2024.

​

DRAVIDIANCODEMIX@FIRE-2024

• This shared task focuses on sarcasm detection in code-mixed text in Dravidian languages.

​

• In this task, there are two languages: Tamil-English and Malayalam-English, and the comments used in this task are collected from social media platforms.

​

• The goal of the proposed shared task is to predict whether the given comments are Sarcastic/Non-Sarcastic. Participants were granted access to development, training, and test datasets.

​

Task Description

II

• The Tamil-English and Malayalam-English dataset contains Social Media comments of code-mixed sentences:
• Eg: -Dai intha padatha eppada release pannuvangada – Sarcastic

-Ellam avan Seyal part 2 .......🤣🤣 Sarcastic

-ஒரு ஓவிய வரைபவன் தன் பிள்ளைக்கு என்ன சொல்லி கொடுப்பான் ஓவியம் வரைய.. என் அப்பன் மிலிட்ரி கோலி விளையாடவா சொல்லி கொடுப்பான்... செம மாஸ் பா சூர்யா அண்ணன்.. Non Sarcastic

​

• Most comments were written in native script and Roman script with either Tamil/Malayalam grammar with English lexicon or English grammar with Tamil/Malayalam lexicon.

​

• The dataset was divided into train, Validation, and test sets.

III

Dataset

III

Dataset

TAMIL-ENGISH

MALAYALAM-ENGISH

CLASS DESCRIPTION

• Tamil-English and Malayalam-English, are the two languages involved in this Shared Task

​

• Methods used to distinguish sarcasm included models like BERT, DistilBERT, XLM-RoBERTa, SVM, Multilingual BERT, IndicBERT, Linear SVC, KNN, ALBERT, MLP Classifier, BiLSTM, etc.

​

• These models were developed by the participating teams to detect sarcasm in the text data.

Methodology

IV

IV

Methodology

MUCS

Text_Catalyst

Change_Makers

UMSNH_NLP

IRLab@IITBHU

- An approach leveraged the strengths of XLM-Roberta, mBERT, Indic-BERT, and Muril to capture the nuanced linguistic features unique to these languages.

-Augmentation done

- F1 scores of 0.74 in Tamil and 0.74 in Malayalam

-They used DistilBERT for detecting sarcasm in Tami

-F1 score of 0.74 for Tamil.

-They used mBERT+CNN model with text augmentation using Contextual Word Embeddings.

-Better than LSTM model trained with Keras embeddings

- F1 score for Tamil of 0.74 and Malayalam of 0.72.

An approach that integrates bag-of-words and deep learning models to create new feature space.

-This feature space is fed into an XGBoost classifier for prediction

-F1 scores of 0.74 in Tamil and 0.76 in Malayalam

-

-They used transformer-based BERT model which is better than random forest, logistic regression, and naive Bayes.

- F1 score of 0.74for Tamil-English

They used BERT with an additional neural network layer for precise classification.

-Observed to better than GPT

-F1 score was 0.74 for Tamil and 0.74 for Malayalam.

IV

Methodology

Sarcasm_NLP

PixelPhrase

JUNLP_Amit Barman

CodeSpark

KEC_Tech_Titan

-They investigated the use of three transformer-based models: (i) DistilBERT, (ii) GoogleBERT, and (iii) RoBERTa

-The F1 score was 0.73 for Tamil and 0.72 for Malayalam.

-hybrid model that combines CNNs, Bi-LSTM networks, and AdaBoost for sarcasm detection.

-F1 score: 0.72 for Malayalam.

-A range of models was utilized, including RoBERTa, CNN, MLP, GRU, RNN, Random Forests (RF), HMM, Logistic Regression, KNN, and GMM.

F1 score: 0.70 for Malayalam.

-

-They have used BERT encoder followed by a classification layer.

-The F1 score : 0.73 for Tamil and 0.72 for Malayalam.

-They have used BiLSTM model with specialized tokenization and embedding

-F1-scores : 0.72 for Tamil and 0.74 for Malayalam.

-hybrid model that combines several neural network architectures and Bi-LSTM networks

-leverages long-range dependencies alongside local feature extraction through the combination of multiple

architectures.

-F1 score: 0.7 for Tamil and 0.67 for Malayalam.

Beyond_Tech

IV

Methodology

Code_Crafters

CJM

MSD

The_Three_Musketeers

KEC_AIDS_79114

- They extracted TF-IDF features that are fed to ML models, Logistic Regression, Multinomial Naive Bayes, and Random Forest Classifier.

- F1 scores of 0.72 in Tamil and 0.62 in Malayalam

-Two sets of embeddings(language-agnostic sentence transformer, LASER encoder pipeline) were concatenated to form the final set of embeddings, which was used to train the MLP classifier.-

-F1 score of 0.68 for Tamil and 0.7 for Malayalam

-ML models like XGBoost, LightGBM, and CatBoost, and deep learning models such as LSTM and GRU.

-SMOTE was applied to address class imbalance

--F1 score of 0.69 for Tamil.

​

- They used TF-IDF followed by logistic regression for prediction

-F1 scores of 0.68 in Tamil and 0.67 in Malayalam

​

-Translates multilingual Tamil-English and Malayalam-English texts into their corresponding English versions, followed by fine-tuning BERT and Xlm-RoBERTa models

-F1 score of 0.68 for Tamil and 0.71 for Malayalam

​

They used used TF-IDF for vectorization followed by logistic regression.

-Observed to better than Decision tree, KNN, SVM

-F1 score was 0.61 for Tamil and 0.58 for Malayalam.

IV

Methodology

TextTitans

Tech_Army_KEC

JUNLP

Tech_Chasers

KEC_AI_InnovationEngineers

-They utilized GPT-3.5 Turbo in zero-shot mode through prompting to carry out sarcasm detection.

-The F1 score was 0.61 for Tamil and 0.5 for Malayalam.

- They built a model using CNN followed by LSTM and AdaBoost classifier to identify

sarcasmF1 score: 0.47 for Tamil.

-They used three ML methods: Logistic Regression, SVC, and Random Forest. Logistic

-F1 score: 0.67 for Tamil.

-

-combined traditional classifiers such as LR, SVM, and Random Forest with advanced methods like CNN, LSTM, and Transformer-based models like BERT and Hierarchical Attention Networks (HAN)

-The F1 score : 0.7for Tamil and 0.67 for Malayalam.

-They have used CNN and BiLSTM model

-F1-scores : 0.5 for Tamil and 0.5 for Malayalam.

​

• 23 teams participated, developed and submitted results in the shared task. All the systems were assessed using the F1-score, harmonic mean of precision and recall which is ideal for balancing both.

​

​

• Transformer-based language models have shown remarkable capabilities by utilizing powerful embedding representations and self-attention mechanisms, advancing the field of language comprehension.

​

​

• In the Tamil-English task, teams “Awsathama", “Team_Catalysts", “Change_Makers", “MUCS", “UMNSH_NLP", “IRLab@IITBHU" took first place achieving a Macro-F1 score of 0.74.

​

​

• For Malayalam-English, UMNSH_NLP" took first place achieving the highest 0.76. Macro-F1 score

V

Result and Discussion

V

Result and Discussion

Rank list for Task: Tamil-English

V

Result and Discussion

Rank list for Task: Malayalam-English

• To summarize this task, Sarcasm Identification in Dravidian languages is used to identify whether the text is sarcastic or Non-Sarcastic.

​

• A total of 23 teams participated in this Sarcasm Identification Shared Task

​

• In this " UMNSH_NLP," the team got the first rank in both Tamil-English and Malayalam-English by achieving a F1 Score of 0.74 and 0.76 respectively.

​

• This is the first shared task on sarcasm detection in Dravidian code-mixed text.

CONCLUSION

VI

Thank you