Evaluating and Comparing ML Algorithms in WEKA

• Course: Databases and Data Mining
• Topic: Model Evaluation & Statistical Testing
• Instructor: Jamolbek Mattiev

Learning Objectives

• • Compare multiple algorithms
• • Evaluate performance on different datasets
• • Perform statistical significance testing
• • Interpret p-values correctly

Why Algorithm Comparison Matters

• • Different algorithms perform differently
• • Dataset characteristics matter
• • Avoid biased conclusions
• • Select best generalizable model

Evaluation Metrics

• • Accuracy
• • Precision
• • Recall
• • F1-score
• • ROC-AUC

Validation Methods in WEKA

• • Percentage split
• • 10-fold cross-validation
• • Repeated cross-validation

Experimental Design

• • Select multiple datasets
• • Apply same validation method
• • Use identical preprocessing
• • Record metrics for each algorithm

Algorithms to Compare (Example)

• • J48 (Decision Tree)
• • Naive Bayes
• • k-Nearest Neighbors
• • Support Vector Machine

Running Experiments in WEKA

• • Open Explorer
• • Load dataset
• • Select Classify tab
• • Choose algorithm and validation

Recording Results

• Create comparison table:
• Dataset | Algorithm | Accuracy | F1 | AUC
• Repeat for multiple datasets

Need for Statistical Testing

• • Differences may occur by chance
• • Must test statistical significance
• • Avoid misleading comparisons

Paired t-Test Concept

• • Compare two algorithms
• • Uses paired results across folds/datasets
• • Null hypothesis: no difference

p-value Interpretation

• • p < 0.05 → statistically significant
• • p ≥ 0.05 → not significant
• • Smaller p → stronger evidence

Using WEKA Experimenter

• • Open Experimenter
• • Add datasets
• • Add algorithms
• • Select cross-validation
• • Run experiment

Performing Significance Test in WEKA

• • Go to Analyse tab
• • Select baseline algorithm
• • Choose statistical test (paired t-test)
• • Interpret output table

Multiple Dataset Comparison

• • Evaluate across many datasets
• • Use average ranks
• • Consider non-parametric tests

Wilcoxon Signed-Rank Test

• • Non-parametric alternative
• • No normality assumption
• • Suitable for multiple datasets

Friedman Test

• • Compare multiple algorithms
• • Rank-based method
• • Followed by post-hoc tests

Common Mistakes

• • Comparing single run results
• • Ignoring variance
• • Misinterpreting p-values

Best Practices

• • Use cross-validation
• • Report mean ± standard deviation
• • Perform statistical testing
• • Compare across multiple datasets

Summary

• • Proper evaluation is essential
• • Statistical testing validates conclusions
• • WEKA Experimenter simplifies comparison
• • Always report reproducible experiments

Algorithm Accuracy Comparison (Example)

Statistical Significance Results (Example)