1
XI INTERNATIONAL CONFERENCE
“INFORMATION TECHNOLOGY AND IMPLEMENTATION” (IT&I-2024)
Evaluation of the Keyword Selection Methods Effectiveness for the Fake News Classification
Khrystyna Lipianina-Honcharenko, Dmytro Lendiuk, Nazar Melnyk, Myroslav Komar, Taras Lendiuk
West Ukrainian National University
2
1. Background of study
The problem of disinformation, in particular the spread of fake news, is gaining more and more importance in the modern information environment. This problem is especially acute in the conditions of hybrid conflicts and information warfare, where false information is used as a tool to influence public opinion and social stability.
This study focuses on the comparison of different keyword selection methods for news classification, such as TF-IDF, RAKE, Yake!, LSA, LDA, and TextRank. The main goal is to identify the most effective approaches that can be used to create a fake news detection tool.
3
2. Research Methodology
Figure 1 illustrates the architecture of the study, which is aimed at a comparative analysis of text processing methods for detecting fake news. At the first stage, data is collected and prepared, where each text document goes through the processes of tokenization, removal of stop words, and bringing the text to the lower case.
4
Figure 1. Structure of the research architecture
5
2.1. Datasets description
Ukrainian News Fake vs True Dataset and Fake News UA Dataset contain news from Ukrainian and Russian sources, collected for the purpose of classifying fake and authentic news.
The Ukrainian News Fake vs True Dataset contains approximately 10,700 news headlines collected from Ukrainian and Russian Telegram channels between February 24 and December 11, 2022, during a full-scale Russian-Ukrainian invasion. Two types of labels are used to classify news: "True" for verified news and "False" for fake news. Data sources include Telegram channels such as Suspilne Novyny, Perepichka NEWS, and NR, as well as disinformation channels, including Vox Ukraine and War on Fakes. The dataset was developed for a university project to classify news and is useful for machine learning research.
6
Figure 2. Class distribution
7
2.2. Description of the used keywords selection
8
9
10
11
2.3. Classifier
12
2.4. Evaluation metrics
Metrics such as precision, recall, F1-score (F-measure) and accuracy (accuracy of classification) are used to evaluate the effectiveness of classification models. Each of these metrics has its own mathematical definition and is used to evaluate various aspects of model performance.
Precision shows the proportion of correct positive predictions among all positive class predictions. This is a metric responsible for the accuracy of predictions regarding positive cases (in our case, for example, fake news). The high accuracy indicates that the model is rarely wrong in classifying news as fake when it is true.
Recall measures the proportion of correctly predicted positive cases among all actual positive cases. This metric is important for evaluating how well the model detects fake news among all available fake news. High completeness means that the model finds the majority of all fake news, but may be wrong in classifying true news as fake.
The F1-score is a harmonic mean between precision and recall and is used to balance these two metrics, especially when it is important not only to predict the correct positive cases, but also to reduce the number of false positives and negatives.
Accuracy is an overall measure of model accuracy, showing the proportion of correct predictions among all predicted cases. It takes into account both correctly predicted positive and negative cases. Accuracy is a useful metric for balanced data, but can be misleading in unbalanced class settings because it can show high accuracy even in cases where one class predominates.
13
3. Results
This section presents the results of a comparative analysis of the effectiveness of different methods of classifying news according to "Fake" or "True". The studied models use different keyword selection approaches, such as TF-IDF, RAKE, Yake!, LSA, LDA, and TextRank. The aim of the analysis was to evaluate the accuracy of the models using precision, recall and F1-score metrics to determine the best algorithms for detecting fake news in textual data.
14
Figure 3. Fake and true news classification report using TF-IDF
15
Figure 4. Fake news and real news classification report using RAKE
16
Figure 5. Fake and true news classification report using Yake!
17
Figure 6. Fake news and real news classification report using LSA
18
Figure 7. Fake and true news classification report using LDA
19
Figure 8. Fake and true news classification report using TextRank
20
Figure 9. Comparative analysis of the accuracy of keyword selection methods for the classification of fake and true news
21
The results of the study showed that the TF-IDF and RAKE methods demonstrated the highest efficiency in news classification, having an accuracy of 0.88. These methods showed a balanced performance for both classes ("Fake" and "True"), especially in terms of precision and f1-score. Other methods, such as LSA, Yake!, TextRank, and LDA, performed worse, particularly in the classification of fake news, which may be due to their lower ability to accurately identify "Fake" signs. The general analysis shows that the TF-IDF and RAKE approaches are the most suitable for more accurate disinformation detection.
22
4. Conclusions
Within the framework of this study, a comparative analysis of keyword selection methods for classifying news into fake and true was conducted. TF-IDF, RAKE, Yake!, LSA, LDA and TextRank methods were used, which were applied to two sets of news data containing headlines from Ukrainian and Russian Telegram channels. The main objective of the study was to compare the performance of these methods using standard classification evaluation metrics such as precision, recall, F1-score and accuracy.
Quantitative results showed that the TF-IDF and RAKE methods became the leaders among all the tested approaches. TF-IDF demonstrated the highest precision of 0.8843, with a high precision for fake news (0.90) and a very high recall for true news (0.94), providing an F1-score of 0.88. The RAKE method performed slightly lower, with an overall accuracy of 0.8794 and an F1-score of 0.85 for fake news and 0.90 for true news. Other methods such as LSA (0.8495), Yake! (0.7805), TextRank (0.7711) and LDA (0.6673), showed worse results.
In the future, it is planned to develop a tool for detecting fake news, which will be based on the results of this study. The tool will integrate the most effective keyword selection methods, including TF-IDF and RAKE, to analyze textual data from news sources.
23
Thank you for attention