Fine-grained Categorization of�Aggressive Texts on Social Media using Weighted�Ensemble of Transformers

Presented By

Omar Sharif

ID No.: 18MCSE001P

Supervised By

Dr. Mohammed Moshiul Hoque

Professor, Dept. of CSE

CUET

1

Department of CSE, CUET

Contents

• Introduction
• Challenges
• Related work
• Research Gap
• Contributions
• Dataset Development
• Baseline Methods
• Proposed Methodology
• Experiments and Results
• Error Analysis
• Conclusion

2

Department of CSE, CUET

Introduction

Aggressive Text Classification: Task of assigning potential texts into predefined aggression categories such as religious, political, verbal, gendered etc.

Necessity,

• Rise of aggressive content in social media
• Incite communal aggression, spread distorted propaganda, damage social harmony
• Ensure the quality of information ecosystem
• Restraining the proliferation of aggressive content has become urgent

3

Department of CSE, CUET

Introduction

Impractical to manually monitor and moderate,

• Massive volume
• Time consuming
• Frequency of information generation
• Cost

Solution,

• Develop automated systems to classify online aggression
• Develop NLP tools to tackle the spread of such undesired content

• Aggressive text identification is still in preliminary stage for low resource languages like Bengali.

4

Department of CSE, CUET

Motivation

• Major motivations,
• No system is developed yet to identify and categorize aggressive Bengali texts into fine-grained classes.
• Such system is required to ensure the quality of conversations and security in the social space
• If we able to detect aggressive texts and identify fine-grained domain of aggression it will help our law enforcement agencies.
• Develop resources and models for Bengali.

5

Department of CSE, CUET

Challenges

• Key barriers,
• Scarcity of benchmark corpora
• Lack of linguistics tools
• Overlapping characteristics with correlated phenomena
• Models trained on other resource-rich languages can not be directly used without proper modifications.
• have to change network architecture (no. of layers, no of neurons etc.)
• tune hyperparameters (learning rate, dropout rate, optimizer etc.)

6

Department of CSE, CUET

Related Work

• Predicting the type and target of offensive posts in social media (Zampieri et al. (2019))
• Compiled a dataset of 14k offensive English post called ‘OLID’.
• Detect offensive language, classify into types and identify targets.
• Baseline evaluation is performed with SVM, CNN, BiLSTM.

​

• Automated hate speech detection and the problem of offensive language. (Davidson et al. (2017))
• Develop a corpus of 25k tweets with three classes (offensive, hate, neither)
• Employed LR, DT, SVM for classification

​

7

Department of CSE, CUET

Related Work

• Hostility detection dataset in Hindi. ( Bhardwaj et al. (2020))
• Multilabel hostility detection dataset of 8.2k posts with five classes (fake, hate, offensive, defamation, non-hostile).
• Baseline system developed with m-BERT embedding and SVM.
• DeepHateExplainer: Explainable hate speech detection in under-resourced Bengali language. (Karim et al. (2021))
• Collected 3k texts and classified them into four hatred classes (political, personal, geopolitical and religious).
• Employed ensemble of BERT-variants to develop their system.
• Threat and abusive language detection on social media in Bengali language. (Chakraborty et al. (2019))
• Applied SVM on a corpus of 5.5k Bengali documents to detect offense and threat

8

Department of CSE, CUET

Summary of the Existing Works in Bengali

• None of the works have been carried out to perform fine-grained categorization of aggressive texts in Bengali.

9

Department of CSE, CUET

Research Gap

• Research questions,
• RQ1: How can we successfully develop an aggression annotated dataset for the Bengali language?
• RQ2: How can we effectively identify potential aggressive texts and categorize them into predefined aggression categories?”.

​

10

Department of CSE, CUET

Contributions

• Develop a novel Bengali Aggressive text Dataset (‘BAD’) with two-level annotation,
• In level-A, 14158 texts are labeled as either aggressive or non-aggressive.
• In level-B, 6807 aggressive texts are categorized into religious, political, verbal and gendered aggression classes each having 2217, 2085, 2043 and 462 texts respectively.
• Develop computational models to identify and perform fine-grained categorization of aggressive Bengali texts.
• Proposed a weighted ensemble technique that can automatically readdress the softmax probabilities of the participating classifiers of ensemble depending on their prior outcomes.

11

Department of CSE, CUET

Contribution 1

Development of Novel Bengali Aggressive Text Dataset

12

Department of CSE, CUET

Definition of the Task

• Applied hierarchical annotation schema to divide ‘BAD’ into two levels:
• (A) Aggressive text identification
• (B) Fine-grained classification of aggressive texts

​

• Level A: Aggressive Text Identification
• Aggressive texts (AG): attack, incite or seek to harm an individual, group or community based on some criteria such as political ideology, religious belief, sexual orientation, gender, race and nationality.
• Non aggressive texts (NoAG): do not contain any statement of aggression or express hidden wish/intent to harm others.

[9] Sharif, O. and Hoque, M.M., “Identification and Classification of Textual Aggression in Social Media: Resource Creation and Evaluation”. Combating Online Hostile Posts in Regional Languages during Emergency Situation, pp.9-20, 2021

13

Department of CSE, CUET

Definition of the Task

• Level B: Classification of Aggressive Text
• Religious Aggression (ReAG): incite violence by attacking religion (Islam, Hindu, Catholic, etc.), religious organizations, or religious belief of a person or a community.
• Political Aggression (PoAG): provoke followers of political parties, condemn political ideology, or excite people in opposition to the state, law or enforcing agencies.
• Verbal Aggression (VeAG): damage social identity and status of the target by using nasty words, curse words and other obscene languages.
• Gendered Aggression (GeAG): promote aggression or attack the victim based on gender, contain aggressive reference to one’s sexual orientation, body parts, sexuality, or other lewd contents.

[9] Sharif, O. and Hoque, M.M., “Identification and Classification of Textual Aggression in Social Media: Resource Creation and Evaluation”. Combating Online Hostile Posts in Regional Languages during Emergency Situation, pp.9-20, 2021

14

Department of CSE, CUET

Dataset Development Steps

Fig 1. Bengali Aggressive Text Dataset development steps

• Collect aggressive and non-aggressive texts from social media platforms.
• Discard duplicate texts and unwanted characters from the raw texts.
• Processed texts pass to the human annotators to carry out manual annotation.

[7] B. Vidgen, L. Derczynski,: Directions in abusive language training data, a systematic review: Garbage in, garbage out, PLOS ONE 15 (12) (2021) 1–32.

15

Department of CSE, CUET

Dataset Development Steps

Table 1. Statistics of few sources from where data were gathered

• Data were collected within the duration from July 2020 to February 2021
• Potential texts are collected from Facebook pages and YouTube channels

16

Department of CSE, CUET

Dataset Development Steps

Fig 2. Annotation guidelines

• To ensure the quality of annotation uniform guideline is followed
• Every text is manually labelled by two annotators
• In case of disagreement, expert resolve the issue through discussion with annotators

17

Department of CSE, CUET

Dataset Development Steps

• Final labelled is assigned following the steps illustrated in the algorithm.

18

Department of CSE, CUET

Dataset Statistics

• To check the validity and quality of annotations kappa score is measured.
• Obtained 0.80 and 0.65 kappa score in coarse-grained and fine-grained labels
• Scores indicate substantial agreement between the annotators in each label

Table 2. Kappa score on each level of annotation

19

Department of CSE, CUET

Dataset Statistics

Table 3. Summary of the train, validation and test set

• ‘BAD’ partitioned into 3 mutually exclusive sets: train, validation, test
• Model is trained on the training set examples
• We used validation set to tweak the model hyperparameters
• Finally model is evaluated on the unseen instances of test set

20

Department of CSE, CUET

Dataset Statistics

Table 4. Statistics of the training set. Here MTL, ANW, ANUW stands for maximum text length, average number of words and average number of unique words respectively

• Training set is highly imbalanced for both coarse and fine grained classes
• NoAG has a total 160k samples while AG ha only 78k samples
• Aggressive texts are shorter in length than the non-aggressive texts
• VeAG has least number of words/text on average

21

Department of CSE, CUET

Dataset Statistics

Table 5:Some examples of BAD. Level-A and level-B indicates coarse and fine-grained class labels.

​

[9] Sharif, O. and Hoque, M.M., “Identification and Classification of Textual Aggression in Social Media: Resource Creation and Evaluation”. Combating Online Hostile Posts in Regional Languages during Emergency Situation, pp.9-20, 2021.

[16] Sharif, O. and Hoque, M.M., “Tackling Cyber-Aggression: Identification and Fine-Grained Categorization of Aggressive Texts on Social Media using Weighted Ensemble of Transformers”. Neurocomputing, vol 474, pp. 82-102, 2021.

22

Department of CSE, CUET

Contribution 2

Development of Computational Models to Perform Fine-grained Categorization

23

Department of CSE, CUET

Baseline Methods

Fig 3. Abstract process diagram of Bengali aggressive text identification and categorization system.

BAD

24

Department of CSE, CUET

Baseline Methods

Fig 4. Schematic Diagram of Baseline Systems

Processed Texts

Input Texts

Feature Extraction

Preprocessing

LR (‘lbfgs’, ‘l2’, c=1.0)

RF (100, ‘gini’)

NB ( x=1.0)

SVM (c=0.5)

Classifier Model

Prediction

Approach 1

Approach 2

Approach 3

(Uni, Bigram)+Tf-idf

CNN

🡪 2 layer

🡪 Filters = 128, 64

🡪 Kernel size = 3

🡪 Pooling type = max

🡪 Window size = 3

Approach 4

BiLSTM

🡪 2 layers

🡪 LSTM cell = 128, 64

🡪 Dropout rate = 0.2

Approach 5

CNN+BiLSTM

🡪 1 layer CNN

🡪 Filters 128, Kernel = 3

🡪 2 LSTM Layer= 64, 32

🡪 Dropout rate = 0.2

Transformers

(4 BERT models)

🡪 learning rate 2e-5

🡪 epochs(20)

🡪 batch size 12

🡪auto_fit method

25

Department of CSE, CUET

Hyperparameters

Table 6. Parameter summary of machine learning models

• ‘lbfgs’ optimizer and ‘l2’ regularization technique used in LR
• SVM is implemented with ‘rbf’ kernel with tolerance value of 0.001
• In RF 100 estimators is used and quality of split is measured with ‘gini’ criterion

26

Department of CSE, CUET

Hyperparameters

Table 7. Hyperparameter summary of deep learning models

• Optimum hyperparameter values have been settled in a trial and error fashion depending on the validation set outcomes.

27

Department of CSE, CUET

Hyperparameters

Table 8. Fine-tuned parameter values of transformers

• All the transformer models fine-tuned using ktrain auto-fit method
• Models are trained for 20 epochs with a batch size of 12 and learning rate 2e−5
• Model weights are stored in the checkpoint.

28

Department of CSE, CUET

Ensemble Approach

• Ensemble of transformers can significantly improve the efficiency of the classifier
• This technique exploits the strength of individual classifiers to form an strong classifier
• This work employs two types of ensemble technique,
• Average ensemble (A-ensemble)
• Weighted ensemble (W-ensemble)

29

Department of CSE, CUET

A-Ensemble

Compute the average of the softmax prob

Output: Class with max prob

• Prior results of the participating models are not considered

BAD

​

30

Department of CSE, CUET

Contribution 3

Proposed Weighted Ensemble Approach

31

Department of CSE, CUET

W-Ensemble

Offer additional weights to the participating models based on prior results

*

*

*

*

Readdressed probability

• This weights helps to dynamically readdress the softmax probabilities

BAD

​

32

Department of CSE, CUET

W-Ensemble

Classifier 1

Classifier 2

c1

c1

c2

c2

Softmax Probabilities

Class 1

Output

• A-Ensemble: prior results of the participating models are not considered

(Acc. 90%)

(Acc. 86%)

33

Department of CSE, CUET

W-Ensemble

Classifier 1

Classifier 2

c1

c1

c2

c2

Readdressed Prob.

Class 2

Output

(Acc. 90%)

(Acc. 86%)

c1

c1

c2

c2

3

2

*

*

Sample weights

Softmax Prob.

34

Department of CSE, CUET

W-Ensemble

• Output of the W-ensemble computed by following equation,

Output Class

Softmax Probabilities

Weights

Normalization

Output: Class with max prob

Probability aggregation

35

Department of CSE, CUET

W-Ensemble

• Algorithm 1 describe the process of calculating ensemble weights,

36

Department of CSE, CUET

Experiments and Results

• Experimental Setup,
• GPU facilitated Google Colab platform (12.5GB RAM,110GB disk space)
• Pandas (1.1.4) and numpy (1.18.5) to process and prepare the data.
• Machine learning models built with scikit-learn packages (0.22.2)
• Deep learning model are trained using Keras (2.4.0) and TensorFlow (2.3.0)
• Transformer are developed using Ktrain (0.25) pacakages.

37

Department of CSE, CUET

Experiments and Results

38

Department of CSE, CUET

Experiments and Results

Table 9. Evaluation results of base classifiers for coarse-grained classification

• LR get maximum f1-score of 0.8968 and CNN get 0.9110 respectively
• All the transformer model achieved higher accuracy than ML, DL methods

39

Department of CSE, CUET

Experiments and Results

Table 10. Evaluation results of ensemble models for coarse-grained classification

• Based on weighted F1-score four base models selected for ensemble
• All possible combinations of the base classifiers are experimented
• W-ensemble of four transformers achieved the highest score of 0.9343.

40

Department of CSE, CUET

Experiments and Results

Table 11. Evaluation results of base classifiers for fine-grained classification

• LR get the maximal f1-score of 0.8689 and CNN+BiLSTM get 0.8691
• Bangla-BERT obtained the highest score of 0.9146 and outperformed all other methods

41

Department of CSE, CUET

Experiments and Results

Table 12. Evaluation results of ensemble models for fine-grained classification

• W-ensemble of four transformers achieved the highest score of 0.9311.

42

Department of CSE, CUET

Error Analysis

Fig. 4. Confusion matrix of the proposed model for coarse and fine-grained classes

• 38 samples are wrongly classifier as NoAG and 55 texts are misclassified as AG.
• Error rate in GeAG class is highest, incorrectly classifies 17 out of 46 texts
• Difficult to identify aggressive texts that contains aggressive references implicitly or sarcastically

43

Department of CSE, CUET

Error Analysis

Table 13. Few examples that are incorrectly classified by the proposed weighted ensemble model

Possible reasons of misclassification,

• Implicit propagation of aggressive texts, Sarcastic use of aggressive words
• Presence of frequent words in both AG and NoAG classes

44

Department of CSE, CUET

�Comparison with Existing Works �

• For consistency, previous methods implemented on the developed dataset and compared with the proposed technique.
• Proposed technique acquired the highest weighted f1-score of 93.43% and 93.11%

45

Department of CSE, CUET

Implications of Proposed W-Ensemble

Implemented on 3 different domains

• Emotion Categorization
• Sentiment Analysis
• Multimodal Meme Analysis

46

Department of CSE, CUET

Emotion Categorization

[4] Tanzia, P., Sharif, O. and Hoque, M.M., “Multi-class Textual Emotion Categorization using Ensemble of Convolutional and Recurrent Neural Network”. SN Computer Science, Vol 3, pp. 1-10, 2021

Table 14. Comparison with existing technique on that work

• Muti-class Bengali emotion categorization: (anger, happy, sad, fear, disgust, joy)
• Experimented on two Bengali emotion dataset

47

Department of CSE, CUET

Sentiment Analysis

[5] Mamun, M.M., P., Sharif, O. and Hoque, M.M., “Classification of Textual Sentiment using Ensemble Technique”. SN Computer Science, Vol 3, pp. 1-13, 2021.

Table 15. Comparison with existing technique on that work

• Bengali sentiment categorization: (positive, negate neutral)
• Experimented on three Bengali sentiment analysis dataset

48

Department of CSE, CUET

Multimodal Meme Detection

• Hossain, E., Sharif, O. and Hoque, M.M. et al., 2021. Identification of Multilingual Offense and Troll from Social Media Memes using Weighted Ensemble of Multimodal Features. (Submission in process)

Table 16. Comparison with existing technique on that work

• Identify offense and troll from multimodal multilingual meme
• Experimented on Tamil and Malayalam dataset

49

Department of CSE, CUET

Conclusion

• Developed ‘BAD’ using hierarchical annotation schema
• Performed baseline evaluation
• ML models: (LR, RF, NB, SVM)
• DL model: (CNN, BiLSTM, CNN+BiLSTM)
• Transformers: (m-BERT, distil-BERT, Bangla-BERT, XLM-R)
• Proposed W-Ensemble and achieved maximum weighted f1-score
• 93.43 % (coarse-grained classes), 93.11 % (fine-grained classes)

Further improvements:

• Identify mixed aggression
• Consider more aggression classes (i.e. racial aggression)
• Add more diverse data in the corpus

50

Department of CSE, CUET

Acknowledgement

This work is supported by ICT Innovation Fund, ICT Division and Directorate of Research and Extension (DRE), Chittagong University of Engineering & Technology.

51

Department of CSE, CUET

References

1. Zampieri, M., Malmasi, S., Nakov, P., Rosenthal, S., Farra, N., Kumar, R.: Predicting the type and target of offensive posts in social media. arXiv preprint arXiv:1902.09666 (2019).
2. Davidson, T., Warmsley, D., Macy, M., Weber, I.: Automated hate speech detection and the problem of offensive language. arXiv preprint arXiv:1703.04009 (2017).
3. M. Bhardwaj, M. S. Akhtar, A. Ekbal, A. Das, and T. Chakraborty.: Hostility detection dataset in hindi. arXiv preprint arXiv:2003.04010 (2020)
4. M. R. Karim, S. K. Dey, and B. R. Chakravarthi.: Deephateexplainer: Explainable hate speech detection in under-resourced bengali language. arXiv preprint arXiv:2103.04010 (2021).
5. Chakraborty, P., Seddiqui, M.H.: Threat and abusive language detection on social media in Bengali language. In: 2019 1st International Conference on Advances in Science, Engineering and Robotics Technology (ICASERT). pp. 1–6. IEEE (2019).
6. P. Fortuna, J. Soler-Company, L. Wanner.: How well do hate speech, toxicity, abusive and offensive language classification models generalize across datasets?, Information Processing & Management 58 (3) (2021).
7. B. Vidgen, L. Derczynski,: Directions in abusive language training data, a systematic review: Garbage in, garbage out, PLOS ONE 15 (12) (2021) 1–32.
8. P. Kapil, A. Ekbal, :A deep neural network based multi-task learning approach to hate speech detection, Knowledge-Based Systems 210 (2020) 106458.
9. Sharif, O. and Hoque, M.M., 2021. Identification and Classification of Textual Aggression in Social Media: Resource Creation and Evaluation. Combating Online Hostile Posts in Regional Languages during Emergency Situation, pp.9-20.

52

Department of CSE, CUET

References

10. Ranasinghe, T. and Zampieri, M., 2020, November. Multilingual Offensive Language Identification with Cross-lingual Embeddings. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) (pp. 5838-5844)
11. Kumari, K. and Singh, J.P., 2020, May. AI_ML_NIT_Patna@ TRAC-2: Deep Learning Approach for Multi-lingual Aggression Identification. In Proceedings of the Second Workshop on Trolling, Aggression and Cyberbullying (pp. 113-119).
12. Baruah, A., Das, K., Barbhuiya, F. and Dey, K., 2020, May. Aggression identification in english, hindi and bangla text using bert, roberta and svm. In Proceedings of the second workshop on trolling, aggression and cyberbullying (pp. 76-82).
13. Nayel, H., 2020, December. NAYEL at SemEval-2020 Task 12: TF/IDF-Based Approach for Automatic Offensive Language Detection in Arabic Tweets. In Proceedings of the Fourteenth Workshop on Semantic Evaluation (pp. 2086-2089).
14. Sharif, O., Hossain, E. and Hoque, M.M., 2021. Offensive Language Detection from Multilingual Code-Mixed Text using Transformers. In Proceedings of the First Workshop on Speech and Language Technologies for Dravidian Languages (pp. 255-261)
15. Hossain, E., Sharif, O. and Hoque, M.M., 2021. Investigating Visual and Textual Features to Identify Trolls from Multimodal Social Media Memes. In Proceedings of the First Workshop on Speech and Language Technologies for Dravidian Languages (pp. 300-306).
16. Sharif, O. and Hoque, M.M., 2021. Tackling Cyber-Aggression: Identification and Fine-Grained Categorization of Aggressive Texts on Social Media using Weighted Ensemble of Transformers. (Submitted to Neurocomputing)

​

53

Department of CSE, CUET

Related Publications

1. Sharif, O. and Hoque, M.M., “Tackling Cyber-Aggression: Identification and Fine-Grained Categorization of Aggressive Texts on Social Media using Weighted Ensemble of Transformers”. Neurocomputing, Vol 474, pp.82-102, 2021.
2. Sharif, O. and Hoque, M.M., “Align and Conquer: An Ensemble Approach to Classify Aggressive Texts from Social Media”. In Proceedings of the International Conference on Signal Processing, Information, Communication and Systems (SPICSCON), 2021.
3. Sharif, O. and Hoque, M.M., “Identification and Classification of Textual Aggression in Social Media: Resource Creation and Evaluation”. In Proceedings of the Combating Online Hostile Posts in Regional Languages during Emergency Situation, pp. 9-20, 2021.
4. Tanzia, P., Sharif, O. and Hoque, M.M., “Multi-class Textual Emotion Categorization using Ensemble of Convolutional and Recurrent Neural Network”. SN Computer Science, Vol 3, pp. 1-10, 2021.
5. Mamun, M.M., P., Sharif, O. and Hoque, M.M., “Classification of Textual Sentiment using Ensemble Technique”. SN Computer Science, Vol 3, pp. 1-13, 2021.

54

Department of CSE, CUET

Related Publications

​

6. Sharif, O., Hossain, E. and Hoque, M.M., “Offensive Language Detection from Multilingual Code-Mixed Text using Transformers”. In Proceedings of the First Workshop on Speech and Language Technologies for Dravidian Languages, pp. 255-261, 2021.
7. Hossain, E., Sharif, O. and Hoque, M.M., “Investigating Visual and Textual Features to Identify Trolls from Multimodal Social Media Memes”. In Proceedings of the First Workshop on Speech and Language Technologies for Dravidian Languages, pp. 300-306, 2021.

55

Department of CSE, CUET

Q & A

56

Department of CSE, CUET