1 of 15

VaxiBERT: A BERT-Based Classifier for Vaccine Tweets with Multi-Label Annotations

FIRE AISoMe 2023

https://sites.google.com/view/aisome/aisome

2 of 15

TEAM: DSIRE

Shivangi Bithel

Samidha Verma

Prachi

Rajat Singh

Data Science and Information Retrieval Lab

Indian Institute of Technology, New Delhi

3 of 15

MOTIVATION

Vaccines have been proven to be a very safe and effective way to fight and eradicate infectious diseases by providing immunity to people to fight against viruses.
However, many people are skeptical about the use of vaccines owing to various reasons, including the politics involved, potential side-effects of vaccines, etc.
Government and various health organizations like WHO can use the insights from micro-blogs to plan their future strategies and encourage everyone to get fully vaccinated.

4 of 15

PROBLEM STATEMENT

TASK: Tweet Classification

Build an effective multi-label classifier to label a social media post (a tweet) according to the specific concern(s) towards vaccines as expressed by the author of the post.

The concerns are described below:

Unnecessary: The tweet indicates vaccines are unnecessary, or that alternate cures are better.
Mandatory: Against mandatory vaccination — The tweet suggests that vaccines should not be made mandatory.
Pharma: Against Big Pharma — The tweet indicates that the Big Pharmaceutical companies are just trying to earn money, or the tweet is against such companies in general because of their history.

5 of 15

Conspiracy: Deeper Conspiracy — The tweet suggests some deeper conspiracy, and not just that the Big Pharma want to make money (e.g., vaccines are being used to track people, COVID is a hoax)
Political: Political side of vaccines — The tweet expresses concerns that the governments / politicians are pushing their own agenda though the vaccines.
Country: Country of origin — The tweet is against some vaccine because of the country where it was developed / manufactured
Rushed: Untested / Rushed Process — The tweet expresses concerns that the vaccines have not been tested properly or that the published data is not accurate.
Ingredients: Vaccine Ingredients / technology — The tweet expresses concerns about the ingredients present in the vaccines (eg. fetal cells, chemicals) or the technology used (e.g., mRNA vaccines can change your DNA)
Side-effect: Side Effects / Deaths — The tweet expresses concerns about the side effects of the vaccines, including deaths caused.
Ineffective: Vaccine is ineffective — The tweet expresses concerns that the vaccines are not effective enough and are useless.
Religious: Religious Reasons — The tweet is against vaccines because of religious reasons
None: No specific reason stated in the tweet, or some reason other than the given ones.

6 of 15

DATASET

Training Data - 9,921 Tweets criticizing COVID-19 vaccine and labeled with one or more of the previously described labels by human annotators

Testing Data - 500 Tweets annotated similarly by human annotators

CAVES: A dataset to facilitate explainable classification and summarization of concerns towards COVID vaccines (in Proc. SIGIR'22)

7 of 15

PRE-PROCESSING

Remove stop words
Convert words to lower-case
Convert emoticons to words
Expand contractions to text
Remove non-alphanumeric characters
Remove URLs

[1] S. Bithel, S. S. Malagi, Unsupervised identification of relevant prior cases, 2021. arXiv:2107.08973. (2021)

[2] S. Bithel, CTC: Covid-19 Tweet Classification using CT-BERT (FIRE IRMiDis 2022)

Tweets generally contain unique lexicons like HASHTAGS, @USER, HTTP-URL and EMOJIS which without pre-processing, often reduce the performance of the model. Thus, we used the following data cleaning pipeline as part of pre-processing the tweets in the dataset:

Stop words like "the", "a", "an", "in", do not provide any valuable information thus we removed them in order to give more focus to the important information.
All words are converted to lower case.
Emojis are extensively used on Twitter to express feelings and emotions. Completely removing them removes a lot of sentiment information; thus, we converted the emojis to text and retained their meanings.
Contractions are converted to its expanded, original form.
We removed all the non-letter characters like brackets, colon, semi-colon, @, etc.
URLs do not help in sentiment analysis; thus, we removed them with the help of regular expression from the text.

8 of 15

EXPERIMENTAL SETUP 1 - COVID TWITTER BERT

The preprocessed dataset is split into training and validation set in the ratio of 80:20.
Used pre-trained COVID-Twitter-BERT (CT-BERT) from Hugging Face and fine tuned it on the training set.
CT-BERT is a domain-specific transformer-based model, pre-trained on a large corpus of 160M tweets posted between January 12 to April 16, 2020, on the topic of COVID-19.

[1] M. Müller, M. Salathé, P. E. Kummervold, Covid-twitter-bert: A natural language processing

model to analyse covid-19 content on twitter, arXiv preprint arXiv:2005.07503 (2020).

9 of 15

EXPERIMENTAL SETUP 2 - OPEN LLaMa 7B (Fine-tuned with pre-processed data)

We used the Prefix Tuning technique, which falls into the larger category of PEFT (Parameter Efficient Fine Tuning) approaches to fine-tune the OpenLLaMa 7B model.
We use 10 extra learnable prompt tokens in our setting and append them to the top 30 transformer layers. This adds only an extra 1.2M parameters over the existing 7B frozen parameters, requiring 10 minutes to train for 9,921 data points using batch size 4, 512 as max sequence length for 5 epochs using a learning rate of 9e-3.

Llama: Open and efficient foundation language models, ArXiv abs/2302.13971 (2023).

10 of 15

Prompt:

Below is an instruction that describes a task, paired with an input that provides further context. Write a response that appropriately satisfy the instruction.

Instruction: The task is multilabel classification where given Input can have one or more than one labels as response. The input tweet belongs to the following labels separated by semicolon - Unnecessary ; Mandatory ; Pharma ; Conspiracy ; Political ; Country ; Rushed ; Ingredients ; Side-effect ; Ineffective ; Religious ; None . These labels can be described as:

Unnecessary: The tweet indicates vaccines are unnecessary, or that alternate cures are better.

Mandatory: Against mandatory vaccination — The tweet suggests that vaccines should not be made mandatory.

Pharma: Against Big Pharma — The tweet indicates that the Big Pharmaceutical companies are just trying to earn money, or the tweet is against such companies in general because of their history.

Conspiracy: Deeper Conspiracy — The tweet suggests some deeper conspiracy, and not just that the Big Pharma want to make money (e.g., vaccines are being used to track people, COVID is a hoax)

Political: Political side of vaccines — The tweet expresses concerns that the governments / politicians are pushing their own agenda though the vaccines.

Country: Country of origin — The tweet is against some vaccine because of the country where it was developed / manufactured

Rushed: Untested / Rushed Process — The tweet expresses concerns that the vaccines have not been tested properly or that the published data is not accurate.

Ingredients: Vaccine Ingredients / technology — The tweet expresses concerns about the ingredients present in the vaccines (eg. fetal cells, chemicals) or the technology used (e.g., mRNA vaccines can change your DNA)

Side-effect: Side Effects / Deaths — The tweet expresses concerns about the side effects of the vaccines, including deaths caused.

Ineffective: Vaccine is ineffective — The tweet expresses concerns that the vaccines are not effective enough and are useless.

Religious: Religious Reasons — The tweet is against vaccines because of religious reasons

None: No specific reason stated in the tweet, or some reason other than the given ones.

Classify the input tweet with semicolon separated labels.

Input: <tweet>

Response: <label_1> ; <label_2> ; ...

Llama: Open and efficient foundation language models, ArXiv abs/2302.13971 (2023).

11 of 15

EXPERIMENTAL SETUP 3 - OPEN LLaMa 7B (Fine-tuned without pre-processed data)

Details of model training are similar to the Run2 except that we didn’t pre-processed the training data with the steps described previously.

[1] H. Touvron, T. Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix, B. Rozière, N. Goyal, E. Hambro, F. Azhar, A. Rodriguez, A. Joulin, E. Grave, G. Lample, Llama: Open and efficient foundation language models, ArXiv abs/2302.13971 (2023). URL: https://api.semanticscholar.org/CorpusID:257219404.

12 of 15

Effectiveness of Pre-processing data on LLM

Sr No.	Method	Macro-F1 Score
1	OpenLLaMa-7B with Pre-processing	0.57
2	OpenLLaMa-7B without Pre-processing	0.55

13 of 15

TEST RESULTS

Sr No.	Team_ID	macro-F1 score	Jaccard Score	Rank
Run 1	DSIRE	0.67	0.70	4
Run 2	DSIRE	0.57	0.61	13
Run 3	DSIRE	0.55	0.60	16

14 of 15

CONCLUSION and FUTURE WORK

We observed that the fine-tuning a domain-specific transformer-based model (CT-BERT) outperformed the fine-tuned OpenLLaMa-7B-based classifier as it yields better results on test data.

In the future we can try using larger LLM based models like GPT-4, Gemini, etc. with full fine-tuning or more efficient fine-tuning strategies.

15 of 15

THANK YOU