1 of 14

DETECTING

DIGITAL

DECEPTION

A comparative study of rule-based and machine learning

approaches for online misinformation detection

Zandile Dladla 4120815

BSc Honours Computer Science

Supervisor: Mr Mogammad Allie

RULES

compared with

MACHINE

LEARNING

2 of 14

The problem and project aim

A lightweight comparison of language-based detection methods

THE PROBLEM

Online misinformation spreads quickly.

Manual verification struggles to keep up.

PROJECT AIM

Build a working prototype and compare weighted

linguistic rules with TF-IDF Logistic Regression.

Lightweight baseline study. BERT, RoBERTa and LLMs provide context for future work.

3 of 14

What the application must do

Requirements informed the interface and functional tests

01

ENTER

Paste an English statement

02

ANALYSE

Submit it for both methods

03

COMPARE

Read labels and rule indicators

04

UPDATE

Replace the text and repeat

Unusable input receives a clear validation message.

4 of 14

The data behind the comparison

LIAR contains fact-checked US political statements from PolitiFact

10,240

TRAINING

Fit vocabulary and model

1,284

VALIDATION

Compare configurations

1,267

TEST

Final evaluation only

Truthful: true and mostly-true

Misleading: half-true, barely-true, false and pants-fire

Half-true retains misleading content under this conservative binary mapping.

Test set: 818 Misleading and 449 Truthful. Fit TF-IDF on training data only.

5 of 14

One statement, two analysis paths

Procedural Python functions connect the interface and both methods

Original

statement

Clean text

TF-IDF features

Weighted phrase rules

Original text

Logistic Regression

Display both outputs

Final label follows ML

The rule score describes matched indicators. It does not override ML.

6 of 14

Implementation

Python in Colab with NLTK, scikit-learn and Gradio

MACHINE LEARNING

Clean and tokenise text

Remove stop words

Build 11,464 TF-IDF features

Apply Logistic Regression

TF-IDF weights words by their importance

within a statement and the training corpus.

WEIGHTED LINGUISTIC RULES

Search the original text with regex

Conspiracy and deceptive claims: 2 each

Sensational, absolute, certainty: 1 each

Score of 2 or more: Misleading

“conspiracy” + “shocking” = 3 / 7

Rule score: 42.86% of maximum weight

The rule score measures matched indicators. It is not a probability of truth.

7 of 14

Model selection and re-evaluation

LR = Logistic Regression. These are validation results, not test results.

Validation model

Accuracy

Balanced

accuracy

Macro F1

Original unigram LR

68.15%

56.63%

0.5578

Balanced unigram + bigram LR

65.42%

63.11%

0.6236

Macro F1 gives both classes equal weight.

The balanced model leads this class-sensitive comparison.

The application and reported test results retain the original model.

The re-evaluation qualifies the earlier accuracy-based choice.

8 of 14

The working application

Recorded interface output, redrawn for readability

EXAMPLE STATEMENT

Climate change is a fake conspiracy

created to manipulate the global economy.

Final classification: Misleading

ML confidence: 66.82%

Rule score: 28.57%

Detected indicator: Conspiracy Language

9 of 14

Testing and refinement

Input validation, repeatable predictions and sequential response time

INPUT VALIDATION

Before: 1234567890

Misleading, 66.82% model confidence

After: rejected as unusable input

Retests reject empty, whitespace, punctuation,

number-only and URL-only inputs.

PROCESSING TIME

3.0 ms

Mean across 100 sequential calls

Maximum: 12.7 ms

All completed below 3 seconds

Application time excludes browser and network delays. No concurrent-load or security audit.

10 of 14

Final test results

1,267 held-out statements, with no further model tuning

4

additional correct predictions over the majority baseline

A gain of 0.32 percentage points in test accuracy

11 of 14

The model misses many truthful statements

Recall measures the share of each actual class detected correctly

91 / 449

truthful statements

correctly classified

by ML

ML balanced accuracy: 54.82% Macro F1: 0.5285

Rules detected only 6 of 818 misleading statements.

12 of 14

Representative errors

Historical LIAR labels compared with predictions from the original model

FALSE POSITIVE

“Building a wall on the U.S.-Mexico

border will take literally years.”

LIAR: Truthful

Model: Misleading

The duration claim needs external evidence.

Border vocabulary cannot establish veracity.

FALSE NEGATIVE

Claim: no opposing-party Senate has

confirmed a Supreme Court nominee

since 1888. (Paraphrased)

LIAR: Misleading (false)

Model: Truthful

Preprocessing removes the year.

The historical assertion needs evidence.

Examples from report Section 12.7. Explanations are qualitative, not causal feature attributions.

13 of 14

Limitations and responsible use

Training on US political claims limits everyday use

“I love my sister.”

Model output: Misleading with 71.24% model confidence

This personal statement lies outside the training domain.

The system does not retrieve evidence or verify facts.

Automated labels can mislead users. Human verification remains necessary.

Avoid confidential inputs. Security and scalability remain unevaluated.

14 of 14

Conclusion and future work

A complete comparative prototype with documented limitations

ML outperformed the selected rules

The small gain over baseline and low truthful recall

limit practical reliability.

Next steps: class-sensitive model selection, preserve negation and quantities,

and evaluate evidence retrieval on a new evaluation set.

sites.google.com/myuwc.ac.za/misinformation-detection/home

Thank you