A Deep Dive into Multilingual

Hate speech Classification

​

Sai Saketh Aluru, Binny Mathew, Punyajoy Saha, Animesh Mukherjee�Department of Computer Science and Engineering, IIT Kharagpur, India

​

CNeRG

Warning!

The following presentation contains words or phrases that are often considered as offensive and hateful by many.

However this cannot be avoided due to the nature of the work.

Overview

Brief description of our work

Hatespeech and its hazards

• Hate speech is defined as a “direct and serious attack on any protected category of people based on their race, ethnicity, national origin, religion, sex, gender, sexual orientation, disability or disease”

​

​

​

​

�

• Crimes related to hatespeech (e.g., Rohingya genocide, Pittsburgh shooting, etc.) are increasing in number

​

​

Work description

• First large scale analysis of multilingual hatespeech
• Languages - 9 languages: Arabic, English, German, Indonesian, Italian, Polish, Portuguese, Spanish and French
• Models Used
1. MUSE _Embeddings + CNN - GRU
2. Translation + BERT
3. LASER _Embeddings + LR
4. mBERT (Multilingual BERT)
• Different Scenarios - Monolingual and Multilingual settings, considered low and high resource cases

Overall results - Toward a benchmark

• Increasing data ⇒ Better performance.

​

​

​

​

​

​

​

• What signals the models pick? LASER + LR → Keyword based | mBERT → Context based

Further details

​

Github Link: https://github.com/punyajoy/DE-LIMIT

HuggingFace: https://huggingface.co/Hate-speech-CNERG

​

SCAN ME!

Detailed Description

EFFECTS OF HATE SPEECH

• Hate speech is increasingly becoming a concerning issue in several countries.
• The public expression of hate speech promotes the devaluation of minority members.
• Frequent and repetitive exposure to hate speech could increase an individual’s outgroup prejudice

Rohingya Genocide

Christchurch Shooting

Sri Lanka riot

Pittsburg Shooting

Plaguing all platforms

Twitter

👹मुसलमानो को करारा जवाब है हर हिन्दु को शेयर करना चाहिये!!! *😡🚩😠🚩😡आज पता चलेगा कितने हिन्दु एक हो गये है!!!!..........*जागो...हिन्दु.....जागो.....

​

Whatsapp

Plaguing all platforms

Twitter

👹मुसलमानो को करारा जवाब है हर हिन्दु को शेयर करना चाहिये!!! *😡🚩😠🚩😡आज पता चलेगा कितने हिन्दु एक हो गये है!!!!..........*जागो...हिन्दु.....जागो.....

​

Whatsapp

Gab

Our efforts

Spread of hate speech (WebSci’ 19, cscw’ 20)

Explainable hate speech detection (AAAI’ 21)

multilingual hate speech detection (ecml/pkdd’ 20, ACL’ 20)

Trapping hateful users (Hypertext’ 21)

Spread of fear speech (The webconf’ 21)

Counterspeech and its types (ICWSM’ 19)

multilingual hate speech detection

​

• First large scale analysis of multilingual hate speech
• Languages - 9 languages: Arabic, English, German, Indonesian, Italian, Polish, Portuguese, Spanish and French
• Models used
• MUSE _Embeddings + CNN - GRU
• Translation + BERT
• LASER _Embeddings + LR
• mBERT (Multilingual BERT)
• Different scenarios - Monolingual and Multilingual settings, considered low and high resource cases

why is this necessary?

• One of the current issues - majority of the hate speech works are available in English language only.

​

* - based on data from hatespeechdata.com

Related works

• The earlier efforts to build hate speech classifiers used simple methods such as dictionary look up, bag-of-words, etc.
• Recently, complex classification models using deep learning and graph embedding techniques have become popular.
• Zhang et al.^[1] used deep neural network, combining convolutional and gated recurrent networks to improve the results on 6 out of 7 datasets used.
• Research into the multilingual hate speech is relatively new. Some works such as Corazza et al.^[2] have studied hate speech in 3 languages.

[1] Zhang, Ziqi, David Robinson, and Jonathan Tepper. "Detecting hate speech on twitter using a convolution-gru based deep neural network." In European semantic web conference, pp. 745-760. Springer, Cham, 2018.

[2] Corazza, Michele, Stefano Menini, Elena Cabrio, Sara Tonelli, and Serena Villata. "A multilingual evaluation for online hate speech detection." ACM Transactions on Internet Technology (TOIT) 20, no. 2 (2020): 1-22.

DATASET DESCRIPTION

Majority dataset in english

Dataset imbalance in most cases

Experimental Setup

LASER^[1]:

MUSE^[2]:

• For each language, we combine all the datasets and perform stratified train/ validation/ test split in the ratio 70%/10%/20%. We report macro F1-score to measure the performance.
• For sentences, LASER embeddings were used and for words MUSE embeddings were used to generate the multilingual representation of the corpus.

[1] Mikel Artetxe and Holger Schwenk. “Massively multilingual sentence embeddings for zero-shot cross-lingual transfer and beyond”. In: Transactions of the Association for Computational Linguistics 7 (2019), pp. 597–610.

[2] Lample, Guillaume, Alexis Conneau, Marc'Aurelio Ranzato, Ludovic Denoyer, and Hervé Jégou. "Word translation without parallel data." In International Conference on Learning Representations. 2018.

Model Architectures used

CNN - GRU:

​

BERT & mBERT:

​

experiments

1. MUSE + CNN-GRU: For the given input sentence, we first obtain the corresponding MUSE embeddings which are then passed as input to the CNN-GRU model.
2. Translation + BERT: The input sentence is first translated to the English language which are then provided as input to the BERT model.
3. LASER + LR: For the given input sentence, we first obtain the corresponding LASER embeddings which are then passed as input to a simple Logistic Regression (LR) model.
4. mBERT: The input sentence is directly fed to the mBert model.

​

hyperparameters

1. MUSE + CNN-GRU
1. Embedding size: MUSE default
2. Word/sentence: 100
2. Translation + BERT
• Translator: Google Translate API
• BERT token length: 128
3. LASER + LR
• Embedding size: LASER default
4. mBERT
• Token length: 128
5. Batch size: 16, learning rate: 2e-5, 3e-5, 5e-5, eochs: 1-5

​

Results - Monolingual

Results - Monolingual

Results - Monolingual

Results - Monolingual

Results - Multilingual

mBERT

All but one language datasets

Target language dataset (incremental steps)

Training

LASER + LR

Validation & Testing�Target language

Results - Multilingual

mBERT

Results - Multilingual

mBERT

All but one language datasets

Target language dataset (incremental steps)

Training

LASER + LR

Validation & Testing�Target language

Hate speech Benchmarks

Recipes for different languages and resource settings, as obtained in our experiments.

​

Interpretability (examples)

​

​

​

Interpretability analysis of LASER + LR and mBERT using LIME^[3]

[3] Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. “” Why should I trust you?” Explaining the predictions of any classifier”. In: Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining. 2016, pp. 1135–1144.

Interpretability (examples)

​

​

​

Yellow - LASER+LR

​

Green - mBERT

Interpretability analysis of LASER + LR and mBERT using LIME^[5]

[5] Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. “” Why should I trust you?” Explaining the predictions of any classifier”. In: Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining. 2016, pp. 1135–1144.

Interpretability (examples)

​

​

​

Yellow - LASER+LR

​

Green - mBERT

Interpretability analysis of LASER + LR and mBERT using LIME^[5]

[5] Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. “” Why should I trust you?” Explaining the predictions of any classifier”. In: Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining. 2016, pp. 1135–1144.

Interpretability (examples)

​

​

​

Yellow - LASER+LR

​

Green - mBERT

Interpretability analysis of LASER + LR and mBERT using LIME^[5]

[5] Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. “” Why should I trust you?” Explaining the predictions of any classifier”. In: Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining. 2016, pp. 1135–1144.

Interpretability (examples)

​

​

​

Yellow - LASER+LR

​

Green - mBERT

Interpretability analysis of LASER + LR and mBERT using LIME^[5]

[5] Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. “” Why should I trust you?” Explaining the predictions of any classifier”. In: Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining. 2016, pp. 1135–1144.

Error Analysis - Types of errors

Confounding Factors (CF)�Errors caused when model relies on some irrelevant features like normalized mentions and links.

Annotation Dilemma (AD)�Ambiguous instances, where according to us model predicts correctly but annotators have labelled it wrong.

Hidden Context (HC)�Errors due to model failing to capture context of the post

Abusive Words (AW)�Errors caused due to over dependance of model on abusive words in input.

Error analysis examples

mBERT:

​

LASER + LR:

​

​

Here “parasites”

refers to immigrants

Here “retarded” is used for movie.

NOTE:- For additional examples please check our paper

conclusion

• In this work, we have analyzed multilingual hate speech using datasets from 16 different sources, comprising of 9 different languages.
• We considered various conditions like low and high resource settings and monolingual or multilingual cases for the different languages.
• As per the observations, for low resource cases, LASER+LR is more effective while for high resource cases, mBERT is usually more effective.

Thank you

Github : �https://github.com/punyajoy/DE-LIMIT��HuggingFace : https://huggingface.co/Hate-speech-CNERG��Contact us:�Animesh Mukherjee: animeshm@cse.iitkgp.ac.in�